StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHow to create a generator/iterator with the Python C API?
text
Body
copied!<p>How do I replicate the following Python code with the Python C API?</p> <pre><code>class Sequence(): def __init__(self, max): self.max = max def data(self): i = 0 while i < self.max: yield i i += 1 </code></pre> <p>So far, I have this:</p> <pre><code>#include <Python/Python.h> #include <Python/structmember.h> /* Define a new object class, Sequence. */ typedef struct { PyObject_HEAD size_t max; } SequenceObject; /* Instance variables */ static PyMemberDef Sequence_members[] = { {"max", T_UINT, offsetof(SequenceObject, max), 0, NULL}, {NULL} /* Sentinel */ }; static int Sequence_Init(SequenceObject *self, PyObject *args, PyObject *kwds) { if (!PyArg_ParseTuple(args, "k", &(self->max))) { return -1; } return 0; } static PyObject *Sequence_data(SequenceObject *self, PyObject *args); /* Methods */ static PyMethodDef Sequence_methods[] = { {"data", (PyCFunction)Sequence_data, METH_NOARGS, "sequence.data() -> iterator object\n" "Returns iterator of range [0, sequence.max)."}, {NULL} /* Sentinel */ }; /* Define new object type */ PyTypeObject Sequence_Type = { PyObject_HEAD_INIT(NULL) 0, /* ob_size */ "Sequence", /* tp_name */ sizeof(SequenceObject), /* tp_basicsize */ 0, /* tp_itemsize */ 0, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags*/ "Test generator object", /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ 0, /* tp_methods */ Sequence_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Sequence_init, /* tp_init */ 0, /* tp_alloc */ PyType_GenericNew, /* tp_new */ }; static PyObject *Sequence_data(SequenceObject *self, PyObject *args) { /* Now what? */ } </code></pre> <p>But I'm not sure where to go next. Could anyone offer some suggestions?</p> <h1>Edit</h1> <p>I suppose the main problem I'm having with this is simulating the <code>yield</code> statement. As I understand it, it is a pretty simple-looking, but in reality complex, statement — it creates a generator with its own <code>__iter__()</code> and <code>next()</code> methods which are called automatically. Searching through the docs, it seems to be associated with the <a href="http://docs.python.org/c-api/gen.html" rel="noreferrer">PyGenObject</a>; however, how to create a new instance of this object is unclear. <code>PyGen_New()</code> takes as its argument a <code>PyFrameObject</code>, the only reference to which I can find is <a href="http://docs.python.org/c-api/reflection.html" rel="noreferrer"><code>PyEval_GetFrame()</code></a>, which doesn't seem to be what I want (or am I mistaken?). Does anyone have any experience with this they can share?</p> <h1>Further Edit</h1> <p>I found this to be clearer when I (essentially) expanded what Python was doing behind the scenes:</p> <pre><code>class IterObject(): def __init__(self, max): self.max = max def __iter__(self): self.i = 0 return self def next(self): if self.i >= self.max: raise StopIteration self.i += 1 return self.i class Sequence(): def __init__(self, max): self.max = max def data(self): return IterObject(self.max) </code></pre> <p>Technically the sequence is off by one but you get the idea.</p> <p>The only problem with this is it's very annoying to create a new object every time one needs a generator — even more so in Python than C because of the required monstrosity that comes with defining a new type. And there can be no <code>yield</code> statement in C because C has no closures. What I did instead (since I couldn't find it in the Python API — <em>please</em> point me to a standard object if it already exists!) was create a simple, generic generator object class that called back a C function for every <code>next()</code> method call. Here it is (note that I have not yet tried compiling this because it is not complete — see below):</p> <pre><code>#include <Python/Python.h> #include <Python/structmember.h> #include <stdlib.h> /* A convenient, generic generator object. */ typedef PyObject *(*callback)(PyObject *callee, void *info) PyGeneratorCallback; typedef struct { PyObject HEAD PyGeneratorCallback callback; PyObject *callee; void *callbackInfo; /* info to be passed along to callback function. */ bool freeInfo; /* true if |callbackInfo| should be free'()d when object * dealloc's, false if not. */ } GeneratorObject; static PyObject *Generator_iter(PyObject *self, PyObject *args) { Py_INCREF(self); return self; } static PyObject *Generator_next(PyObject *self, PyObject *args) { return self->callback(self->callee, self->callbackInfo); } static PyMethodDef Generator_methods[] = { {"__iter__", (PyCFunction)Generator_iter, METH_NOARGS, NULL}, {"next", (PyCFunction)Generator_next, METH_NOARGS, NULL}, {NULL} /* Sentinel */ }; static void Generator_dealloc(GenericEventObject *self) { if (self->freeInfo && self->callbackInfo != NULL) { free(self->callbackInfo); } self->ob_type->tp_free((PyObject *)self); } PyTypeObject Generator_Type = { PyObject_HEAD_INIT(NULL) 0, /* ob_size */ "Generator", /* tp_name */ sizeof(GeneratorObject), /* tp_basicsize */ 0, /* tp_itemsize */ Generator_dealloc, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags*/ 0, /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ 0, /* tp_methods */ 0, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ 0, /* tp_alloc */ PyType_GenericNew, /* tp_new */ }; /* Returns a new generator object with the given callback function * and arguments. */ PyObject *Generator_New(PyObject *callee, void *info, bool freeInfo, PyGeneratorCallback callback) { GeneratorObject *generator = (GeneratorObject *)_PyObject_New(&Generator_Type); if (generator == NULL) return NULL; generator->callee = callee; generator->info = info; generator->callback = callback; self->freeInfo = freeInfo; return (PyObject *)generator; } /* End of Generator definition. */ /* Define a new object class, Sequence. */ typedef struct { PyObject_HEAD size_t max; } SequenceObject; /* Instance variables */ static PyMemberDef Sequence_members[] = { {"max", T_UINT, offsetof(SequenceObject, max), 0, NULL}, {NULL} /* Sentinel */ } static int Sequence_Init(SequenceObject *self, PyObject *args, PyObject *kwds) { if (!PyArg_ParseTuple(args, "k", &self->max)) { return -1; } return 0; } static PyObject *Sequence_data(SequenceObject *self, PyObject *args); /* Methods */ static PyMethodDef Sequence_methods[] = { {"data", (PyCFunction)Sequence_data, METH_NOARGS, "sequence.data() -> iterator object\n" "Returns generator of range [0, sequence.max)."}, {NULL} /* Sentinel */ }; /* Define new object type */ PyTypeObject Sequence_Type = { PyObject_HEAD_INIT(NULL) 0, /* ob_size */ "Sequence", /* tp_name */ sizeof(SequenceObject), /* tp_basicsize */ 0, /* tp_itemsize */ 0, /* tp_dealloc */ 0, /* tp_print */ 0, /* tp_getattr */ 0, /* tp_setattr */ 0, /* tp_compare */ 0, /* tp_repr */ 0, /* tp_as_number */ 0, /* tp_as_sequence */ 0, /* tp_as_mapping */ 0, /* tp_hash */ 0, /* tp_call */ 0, /* tp_str */ 0, /* tp_getattro */ 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags*/ "Test generator object", /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ 0, /* tp_weaklistoffset */ 0, /* tp_iter */ 0, /* tp_iternext */ 0, /* tp_methods */ Sequence_members, /* tp_members */ 0, /* tp_getset */ 0, /* tp_base */ 0, /* tp_dict */ 0, /* tp_descr_get */ 0, /* tp_descr_set */ 0, /* tp_dictoffset */ (initproc)Sequence_init, /* tp_init */ 0, /* tp_alloc */ PyType_GenericNew, /* tp_new */ }; static PyObject *Sequence_data(SequenceObject *self, PyObject *args) { size_t *info = malloc(sizeof(size_t)); if (info == NULL) return NULL; *info = 0; /* |info| will be free'()d by the returned generator object. */ GeneratorObject *ret = Generator_New(self, info, true, &Sequence_data_next_callback); if (ret == NULL) { free(info); /* Watch out for memory leaks! */ } return ret; } PyObject *Sequence_data_next_callback(PyObject *self, void *info) { size_t i = info; if (i > self->max) { return NULL; /* TODO: How do I raise StopIteration here? I can't seem to find * a standard exception. */ } else { return Py_BuildValue("k", i++); } } </code></pre> <p>However, unfortunately, I'm still not finished. The only question I have left is: How do I raise a <code>StopIteration</code> exception with the C API? I can't seem to find it listed in the <a href="http://docs.python.org/c-api/exceptions.html#standard-exceptions" rel="noreferrer">Standard Exceptions</a>. Also, perhaps more importantly, is this the correct way to approach this problem?</p> <p>Thanks to anyone that's still following this.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload