Browse Source

python: preserve type in StridedArrayViews created from buffers.

This finally makes it possible to expose APIs that take StridedArrayView
instances as an input, until now the type information was always lost,
making all views plain bytes and thus impossible to check whether the
types passed were a large enough size at least, if nothing else.

Preserving the type means there has to be type-dependent implementation
for __getitem__() and __setitem__(). So far this is only done for the
very basic builtin types, similarly to what Python's own array supports.
next
Vladimír Vondruš 2 years ago
parent
commit
78897dddf7
  1. 91
      doc/python/corrade.containers.rst
  2. 18
      src/Corrade/Containers/StridedArrayViewPythonBindings.h
  3. 70
      src/python/corrade/containers.cpp
  4. 86
      src/python/corrade/test/test_containers.py
  5. 34
      src/python/corrade/test/test_containers_numpy.py

91
doc/python/corrade.containers.rst

@ -25,6 +25,7 @@
.. doctest setup .. doctest setup
>>> from corrade import containers >>> from corrade import containers
>>> import array
.. py:class:: corrade.containers.ArrayView .. py:class:: corrade.containers.ArrayView
@ -82,9 +83,21 @@
.. py:class:: corrade.containers.StridedArrayView1D .. py:class:: corrade.containers.StridedArrayView1D
Provides an one-dimensional read-only view on a memory range with custom Provides a typed one-dimensional read-only view on a memory range with
stride values. Convertible both to and from Python objects supporting the custom stride values. Convertible both to and from Python objects
Buffer Protocol. See :ref:`StridedArrayView2D`, :ref:`StridedArrayView3D`, supporting the Buffer Protocol, preserving the dimensionality, type and
stride information:
.. code:: pycon
>>> a = array.array('f', [2.5, 3.14, -1.75, 53.2])
>>> b = containers.StridedArrayView1D(memoryview(a)[::2])
>>> b[0]
2.5
>>> b[1]
-1.75
See :ref:`StridedArrayView2D`, :ref:`StridedArrayView3D`,
:ref:`StridedArrayView4D`, :ref:`MutableStridedArrayView1D` and others for :ref:`StridedArrayView4D`, :ref:`MutableStridedArrayView1D` and others for
multi-dimensional and mutable equivalents. multi-dimensional and mutable equivalents.
@ -104,6 +117,12 @@
multi-dimensional slicing as well (which raises :ref:`NotImplementedError` multi-dimensional slicing as well (which raises :ref:`NotImplementedError`
in Py3.7 :ref:`memoryview`). in Py3.7 :ref:`memoryview`).
.. py:function:: corrade.containers.StridedArrayView1D.__getitem__(self, i: int)
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <StridedArrayView1D.format>` is not one of :py:`'b'`,
:py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`, :py:`'q'`,
:py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.StridedArrayView1D.flipped .. py:function:: corrade.containers.StridedArrayView1D.flipped
:raise IndexError: if :p:`dimension` is not :py:`0` :raise IndexError: if :p:`dimension` is not :py:`0`
.. py:function:: corrade.containers.StridedArrayView1D.broadcasted .. py:function:: corrade.containers.StridedArrayView1D.broadcasted
@ -118,6 +137,18 @@
Equivalent to :ref:`StridedArrayView1D`, but implementing Equivalent to :ref:`StridedArrayView1D`, but implementing
:ref:`__setitem__()` as well. :ref:`__setitem__()` as well.
.. py:function:: corrade.containers.MutableStridedArrayView1D.__getitem__(self, i: int)
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <MutableStridedArrayView1D.format>` is not one of
:py:`'b'`, :py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`,
:py:`'q'`, :py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.MutableStridedArrayView1D.__setitem__(self, i: int, value: handle)
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <MutableStridedArrayView1D.format>` is not one of
:py:`'b'`, :py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`,
:py:`'q'`, :py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.MutableStridedArrayView1D.flipped .. py:function:: corrade.containers.MutableStridedArrayView1D.flipped
:raise IndexError: if :p:`dimension` is not :py:`0` :raise IndexError: if :p:`dimension` is not :py:`0`
.. py:function:: corrade.containers.MutableStridedArrayView1D.broadcasted .. py:function:: corrade.containers.MutableStridedArrayView1D.broadcasted
@ -131,6 +162,12 @@
See :ref:`StridedArrayView1D` for more information. See :ref:`StridedArrayView1D` for more information.
.. py:function:: corrade.containers.StridedArrayView2D.__getitem__(self, i: typing.Tuple[int, int])
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <StridedArrayView2D.format>` is not one of :py:`'b'`,
:py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`, :py:`'q'`,
:py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.StridedArrayView2D.flipped .. py:function:: corrade.containers.StridedArrayView2D.flipped
:raise IndexError: if :p:`dimension` is not :py:`0` or :py:`1` :raise IndexError: if :p:`dimension` is not :py:`0` or :py:`1`
.. py:function:: corrade.containers.StridedArrayView2D.broadcasted .. py:function:: corrade.containers.StridedArrayView2D.broadcasted
@ -148,6 +185,18 @@
See :ref:`StridedArrayView1D` and :ref:`MutableStridedArrayView1D` for more See :ref:`StridedArrayView1D` and :ref:`MutableStridedArrayView1D` for more
information. information.
.. py:function:: corrade.containers.MutableStridedArrayView2D.__getitem__(self, i: typing.Tuple[int, int])
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <MutableStridedArrayView2D.format>` is not one of
:py:`'b'`, :py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`,
:py:`'q'`, :py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.MutableStridedArrayView2D.__setitem__(self, i: typing.Tuple[int, int], value: handle)
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <MutableStridedArrayView2D.format>` is not one of
:py:`'b'`, :py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`,
:py:`'q'`, :py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.MutableStridedArrayView2D.flipped .. py:function:: corrade.containers.MutableStridedArrayView2D.flipped
:raise IndexError: if :p:`dimension` is not :py:`0` or :py:`1` :raise IndexError: if :p:`dimension` is not :py:`0` or :py:`1`
.. py:function:: corrade.containers.MutableStridedArrayView2D.broadcasted .. py:function:: corrade.containers.MutableStridedArrayView2D.broadcasted
@ -164,6 +213,12 @@
See :ref:`StridedArrayView1D` for more information. See :ref:`StridedArrayView1D` for more information.
.. py:function:: corrade.containers.StridedArrayView3D.__getitem__(self, i: typing.Tuple[int, int, int])
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <StridedArrayView3D.format>` is not one of :py:`'b'`,
:py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`, :py:`'q'`,
:py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.StridedArrayView3D.flipped .. py:function:: corrade.containers.StridedArrayView3D.flipped
:raise IndexError: if :p:`dimension` is not :py:`0`, :py:`1` or :py:`2` :raise IndexError: if :p:`dimension` is not :py:`0`, :py:`1` or :py:`2`
.. py:function:: corrade.containers.StridedArrayView3D.broadcasted .. py:function:: corrade.containers.StridedArrayView3D.broadcasted
@ -181,6 +236,18 @@
See :ref:`StridedArrayView1D` and :ref:`MutableStridedArrayView1D` for more See :ref:`StridedArrayView1D` and :ref:`MutableStridedArrayView1D` for more
information. information.
.. py:function:: corrade.containers.MutableStridedArrayView3D.__getitem__(self, i: typing.Tuple[int, int, int])
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <MutableStridedArrayView3D.format>` is not one of
:py:`'b'`, :py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`,
:py:`'q'`, :py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.MutableStridedArrayView3D.__setitem__(self, i: typing.Tuple[int, int, int], value: handle)
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <MutableStridedArrayView3D.format>` is not one of
:py:`'b'`, :py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`,
:py:`'q'`, :py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.MutableStridedArrayView3D.flipped .. py:function:: corrade.containers.MutableStridedArrayView3D.flipped
:raise IndexError: if :p:`dimension` is not :py:`0`, :py:`1` or :py:`2` :raise IndexError: if :p:`dimension` is not :py:`0`, :py:`1` or :py:`2`
.. py:function:: corrade.containers.MutableStridedArrayView3D.broadcasted .. py:function:: corrade.containers.MutableStridedArrayView3D.broadcasted
@ -197,6 +264,12 @@
See :ref:`StridedArrayView1D` for more information. See :ref:`StridedArrayView1D` for more information.
.. py:function:: corrade.containers.StridedArrayView4D.__getitem__(self, i: typing.Tuple[int, int, int, int])
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <StridedArrayView4D.format>` is not one of :py:`'b'`,
:py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`, :py:`'q'`,
:py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.StridedArrayView4D.flipped .. py:function:: corrade.containers.StridedArrayView4D.flipped
:raise IndexError: if :p:`dimension` is not :py:`0`, :py:`1` :py:`2` or :raise IndexError: if :p:`dimension` is not :py:`0`, :py:`1` :py:`2` or
:py:`3` :py:`3`
@ -212,6 +285,18 @@
See :ref:`StridedArrayView1D` and :ref:`MutableStridedArrayView1D` for more See :ref:`StridedArrayView1D` and :ref:`MutableStridedArrayView1D` for more
information. information.
.. py:function:: corrade.containers.MutableStridedArrayView4D.__getitem__(self, i: typing.Tuple[int, int, int, int])
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <MutableStridedArrayView4D.format>` is not one of
:py:`'b'`, :py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`,
:py:`'q'`, :py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.MutableStridedArrayView4D.__setitem__(self, i: typing.Tuple[int, int, int, int], value: handle)
:raise IndexError: If :p:`i` is out of range
:raise NotImplementedError: If the view was created from a buffer and
:ref:`format <MutableStridedArrayView4D.format>` is not one of
:py:`'b'`, :py:`'B'`, :py:`'h'`, :py:`'H'`, :py:`'i'`, :py:`'I'`,
:py:`'q'`, :py:`'Q'`, :py:`'f'` or :py:`'d'`
.. py:function:: corrade.containers.MutableStridedArrayView4D.flipped .. py:function:: corrade.containers.MutableStridedArrayView4D.flipped
:raise IndexError: if :p:`dimension` is not :py:`0`, :py:`1` :py:`2` or :raise IndexError: if :p:`dimension` is not :py:`0`, :py:`1` :py:`2` or
:py:`3` :py:`3`

18
src/Corrade/Containers/StridedArrayViewPythonBindings.h

@ -27,13 +27,16 @@
#include <pybind11/pybind11.h> #include <pybind11/pybind11.h>
#include <Corrade/Containers/StridedArrayView.h> #include <Corrade/Containers/StridedArrayView.h>
#include <Corrade/Containers/String.h>
namespace Corrade { namespace Containers { namespace Corrade { namespace Containers {
namespace Implementation { namespace Implementation {
/* For maintainability please keep in the same order as /* For maintainability please keep in the same order as
https://docs.python.org/3/library/struct.html#format-characters */ https://docs.python.org/3/library/struct.html#format-characters. Each of
these has also a corresponding entry in accessorsForFormat() in
containers.cpp in the same order. */
template<class T> constexpr const char* pythonFormatString() { template<class T> constexpr const char* pythonFormatString() {
static_assert(sizeof(T) == 0, "format string unknown for this type, supply it explicitly"); static_assert(sizeof(T) == 0, "format string unknown for this type, supply it explicitly");
return {}; return {};
@ -111,7 +114,7 @@ template<unsigned dimensions, class T> class PyStridedArrayView: public StridedA
template<class U> explicit PyStridedArrayView(const StridedArrayView<dimensions, U>& view): PyStridedArrayView{view, Implementation::pythonFormatString<typename std::decay<U>::type>(), sizeof(U)} {} template<class U> explicit PyStridedArrayView(const StridedArrayView<dimensions, U>& view): PyStridedArrayView{view, Implementation::pythonFormatString<typename std::decay<U>::type>(), sizeof(U)} {}
template<class U> explicit PyStridedArrayView(const StridedArrayView<dimensions, U>& view, const char* format, std::size_t itemsize): PyStridedArrayView<dimensions, T>{ template<class U> explicit PyStridedArrayView(const StridedArrayView<dimensions, U>& view, Containers::StringView format, std::size_t itemsize): PyStridedArrayView<dimensions, T>{
arrayCast<T>(view), arrayCast<T>(view),
format, format,
itemsize, itemsize,
@ -119,7 +122,7 @@ template<unsigned dimensions, class T> class PyStridedArrayView: public StridedA
Implementation::PyStridedArrayViewSetItem<T, U>::set Implementation::PyStridedArrayViewSetItem<T, U>::set
} {} } {}
explicit PyStridedArrayView(const StridedArrayView<dimensions, T>& view, const char* format, std::size_t itemsize, pybind11::object(*getitem)(const char*), void(*setitem)(char*, pybind11::handle)): StridedArrayView<dimensions, T>{view}, format{format}, itemsize{itemsize}, getitem{getitem}, setitem{setitem} {} explicit PyStridedArrayView(const StridedArrayView<dimensions, T>& view, Containers::StringView format, std::size_t itemsize, pybind11::object(*getitem)(const char*), void(*setitem)(char*, pybind11::handle)): StridedArrayView<dimensions, T>{view}, format{format}, itemsize{itemsize}, getitem{getitem}, setitem{setitem} {}
/* All APIs that are exposed by bindings and return a StridedArrayView /* All APIs that are exposed by bindings and return a StridedArrayView
have to return the wrapper now */ have to return the wrapper now */
@ -169,7 +172,10 @@ template<unsigned dimensions, class T> class PyStridedArrayView: public StridedA
} }
/* has to be public as it's accessed by the bindings directly */ /* has to be public as it's accessed by the bindings directly */
const char* format; /* The assumption is that >99% of format strings should be just a few
characters, stored with a SSO. I.e., not even bothering with
String::nullTerminatedGlobalView() anywhere. */
Containers::String format;
std::size_t itemsize; std::size_t itemsize;
pybind11::object(*getitem)(const char*); pybind11::object(*getitem)(const char*);
void(*setitem)(char*, pybind11::handle); void(*setitem)(char*, pybind11::handle);
@ -178,13 +184,13 @@ template<unsigned dimensions, class T> class PyStridedArrayView: public StridedA
namespace Implementation { namespace Implementation {
template<unsigned dimensions, class T> struct PyStridedElement { template<unsigned dimensions, class T> struct PyStridedElement {
static PyStridedArrayView<dimensions - 1, T> wrap(const StridedArrayView<dimensions - 1, T>& element, const char* format, std::size_t itemsize, pybind11::object(*getitem)(const char*), void(*setitem)(char*, pybind11::handle)) { static PyStridedArrayView<dimensions - 1, T> wrap(const StridedArrayView<dimensions - 1, T>& element, Containers::StringView format, std::size_t itemsize, pybind11::object(*getitem)(const char*), void(*setitem)(char*, pybind11::handle)) {
return PyStridedArrayView<dimensions - 1, T>{element, format, itemsize, getitem, setitem}; return PyStridedArrayView<dimensions - 1, T>{element, format, itemsize, getitem, setitem};
} }
}; };
template<class T> struct PyStridedElement<1, T> { template<class T> struct PyStridedElement<1, T> {
static T& wrap(T& element, const char*, std::size_t, pybind11::object(*)(const char*), void(*)(char*, pybind11::handle)) { static T& wrap(T& element, Containers::StringView, std::size_t, pybind11::object(*)(const char*), void(*)(char*, pybind11::handle)) {
return element; return element;
} }
}; };

70
src/python/corrade/containers.cpp

@ -28,6 +28,7 @@
#include <Corrade/Containers/Array.h> #include <Corrade/Containers/Array.h>
#include <Corrade/Containers/BitArray.h> #include <Corrade/Containers/BitArray.h>
#include <Corrade/Containers/StridedBitArrayView.h> #include <Corrade/Containers/StridedBitArrayView.h>
#include <Corrade/Containers/Pair.h>
#include <Corrade/Containers/ScopeGuard.h> #include <Corrade/Containers/ScopeGuard.h>
#include "Corrade/Containers/PythonBindings.h" #include "Corrade/Containers/PythonBindings.h"
@ -38,6 +39,8 @@
namespace corrade { namespace corrade {
using namespace Containers::Literals;
namespace { namespace {
struct Slice { struct Slice {
@ -350,7 +353,9 @@ template<class T> bool stridedArrayViewBufferProtocol(T& self, Py_buffer& buffer
buffer.buf = const_cast<typename std::decay<typename T::ErasedType>::type*>(self.data()); buffer.buf = const_cast<typename std::decay<typename T::ErasedType>::type*>(self.data());
buffer.readonly = std::is_const<typename T::Type>::value; buffer.readonly = std::is_const<typename T::Type>::value;
if((flags & PyBUF_FORMAT) == PyBUF_FORMAT) if((flags & PyBUF_FORMAT) == PyBUF_FORMAT)
buffer.format = const_cast<char*>(self.format); /* A valid format shouldn't be an empty string. If it is, it's because
it was null originally. Pass null in that case again. */
buffer.format = self.format ? self.format.data() : nullptr;
/* The view is immutable (can't change its size after it has been /* The view is immutable (can't change its size after it has been
constructed), so referencing the size/stride directly is okay */ constructed), so referencing the size/stride directly is okay */
buffer.shape = const_cast<Py_ssize_t*>(reinterpret_cast<const Py_ssize_t*>(Containers::Implementation::sizeRef(self).begin())); buffer.shape = const_cast<Py_ssize_t*>(reinterpret_cast<const Py_ssize_t*>(Containers::Implementation::sizeRef(self).begin()));
@ -557,6 +562,44 @@ template<unsigned dimensions, template<unsigned> class Steps, class T> Container
return Containers::pyArrayViewHolder(sliced, empty ? py::none{} : std::move(owner)); return Containers::pyArrayViewHolder(sliced, empty ? py::none{} : std::move(owner));
} }
Containers::Pair<py::object(*)(const char*), void(*)(char*, py::handle)> accessorsForFormat(const char* const format) {
/* The format string can be null, in which case B should be assumed:
https://docs.python.org/3/c-api/buffer.html#c.Py_buffer.format */
const Containers::StringView formatString = format ? format : "B"_s;
/* Matching the entries (and order) in StridedArrayViewPythonBindings.h */
#define _c(string, type) \
if(formatString == #string ## _s) return { \
[](const char* item) { \
return py::cast(*reinterpret_cast<const type*>(item)); \
}, \
[](char* item, py::handle object) { \
*reinterpret_cast<type*>(item) = py::cast<type>(object); \
}};
_c(b, std::int8_t)
_c(B, std::uint8_t)
_c(h, std::int16_t)
_c(H, std::uint16_t)
_c(i, std::int32_t)
_c(I, std::uint32_t)
/** @todo numpy's np.int64 is a `l` even though struct says it's 4 bytes,
what to do?! */
_c(q, std::int64_t)
_c(Q, std::uint64_t)
_c(f, float)
_c(d, double)
return {
[](const char*) -> py::object {
PyErr_SetString(PyExc_NotImplementedError, "access to this data format is not implemented, sorry");
throw py::error_already_set{};
},
[](char*, py::handle) {
PyErr_SetString(PyExc_NotImplementedError, "access to this data format is not implemented, sorry");
throw py::error_already_set{};
}};
}
template<unsigned dimensions, class T> void stridedArrayView(py::class_<Containers::PyStridedArrayView<dimensions, T>, Containers::PyArrayViewHolder<Containers::PyStridedArrayView<dimensions, T>>>& c) { template<unsigned dimensions, class T> void stridedArrayView(py::class_<Containers::PyStridedArrayView<dimensions, T>, Containers::PyArrayViewHolder<Containers::PyStridedArrayView<dimensions, T>>>& c) {
/* Implicitly convertible from a buffer */ /* Implicitly convertible from a buffer */
py::implicitly_convertible<py::buffer, Containers::PyStridedArrayView<dimensions, T>>(); py::implicitly_convertible<py::buffer, Containers::PyStridedArrayView<dimensions, T>>();
@ -571,7 +614,7 @@ template<unsigned dimensions, class T> void stridedArrayView(py::class_<Containe
.def(py::init([](const py::buffer& other) { .def(py::init([](const py::buffer& other) {
/* GCC 4.8 otherwise loudly complains about missing initializers */ /* GCC 4.8 otherwise loudly complains about missing initializers */
Py_buffer buffer{nullptr, nullptr, 0, 0, 0, 0, nullptr, nullptr, nullptr, nullptr, nullptr}; Py_buffer buffer{nullptr, nullptr, 0, 0, 0, 0, nullptr, nullptr, nullptr, nullptr, nullptr};
if(PyObject_GetBuffer(other.ptr(), &buffer, PyBUF_STRIDES|(std::is_const<T>::value ? 0 : PyBUF_WRITABLE)) != 0) if(PyObject_GetBuffer(other.ptr(), &buffer, PyBUF_STRIDES|PyBUF_FORMAT|(std::is_const<T>::value ? 0 : PyBUF_WRITABLE)) != 0)
throw py::error_already_set{}; throw py::error_already_set{};
Containers::ScopeGuard e{&buffer, PyBuffer_Release}; Containers::ScopeGuard e{&buffer, PyBuffer_Release};
@ -581,6 +624,8 @@ template<unsigned dimensions, class T> void stridedArrayView(py::class_<Containe
throw py::error_already_set{}; throw py::error_already_set{};
} }
const Containers::Pair<py::object(*)(const char*), void(*)(char*, py::handle)> accessors = accessorsForFormat(buffer.format);
/* Calculate total memory size that spans the whole view. Mainly to /* Calculate total memory size that spans the whole view. Mainly to
make the constructor assert happy, not used otherwise */ make the constructor assert happy, not used otherwise */
std::size_t size = 0; std::size_t size = 0;
@ -592,12 +637,15 @@ template<unsigned dimensions, class T> void stridedArrayView(py::class_<Containe
the buffer because we no longer care about the buffer the buffer because we no longer care about the buffer
descriptor -- that could allow the GC to haul away a bit more descriptor -- that could allow the GC to haul away a bit more
garbage */ garbage */
/** @todo this always assumes bytes for now -- remember the format return Containers::pyArrayViewHolder(Containers::PyStridedArrayView<dimensions, T>{
and provide a checked typed conversion API */ Containers::StridedArrayView<dimensions, T>{
return Containers::pyArrayViewHolder(Containers::PyStridedArrayView<dimensions, T>{Containers::StridedArrayView<dimensions, T>{ {static_cast<T*>(buffer.buf), size},
{static_cast<T*>(buffer.buf), size}, Containers::StaticArrayView<dimensions, const std::size_t>{reinterpret_cast<std::size_t*>(buffer.shape)},
Containers::StaticArrayView<dimensions, const std::size_t>{reinterpret_cast<std::size_t*>(buffer.shape)}, Containers::StaticArrayView<dimensions, const std::ptrdiff_t>{reinterpret_cast<std::ptrdiff_t*>(buffer.strides)}},
Containers::StaticArrayView<dimensions, const std::ptrdiff_t>{reinterpret_cast<std::ptrdiff_t*>(buffer.strides)}}}, buffer.format,
std::size_t(buffer.itemsize),
accessors.first(),
accessors.second()},
buffer.len ? py::reinterpret_borrow<py::object>(buffer.obj) : py::none{}); buffer.len ? py::reinterpret_borrow<py::object>(buffer.obj) : py::none{});
}), "Construct from a buffer") }), "Construct from a buffer")
@ -613,7 +661,11 @@ template<unsigned dimensions, class T> void stridedArrayView(py::class_<Containe
}, "View stride in each dimension") }, "View stride in each dimension")
.def_property_readonly("dimensions", [](const Containers::PyStridedArrayView<dimensions, T>&) { return dimensions; }, "Dimension count") .def_property_readonly("dimensions", [](const Containers::PyStridedArrayView<dimensions, T>&) { return dimensions; }, "Dimension count")
.def_property_readonly("format", [](const Containers::PyStridedArrayView<dimensions, T>& self) { .def_property_readonly("format", [](const Containers::PyStridedArrayView<dimensions, T>& self) {
return self.format; /* A valid format shouldn't be an empty string. If it is, it's
because it was null originally. Return null in that case to
turn this into a None, consistently with how Python's
memoryview etc expects the format strings to be. */
return self.format ? self.format.data() : nullptr;
}, "Format of each item") }, "Format of each item")
.def_property_readonly("owner", [](const Containers::PyStridedArrayView<dimensions, T>& self) { .def_property_readonly("owner", [](const Containers::PyStridedArrayView<dimensions, T>& self) {
return pyObjectHolderFor<Containers::PyArrayViewHolder>(self).owner; return pyObjectHolderFor<Containers::PyArrayViewHolder>(self).owner;

86
src/python/corrade/test/test_containers.py

@ -275,9 +275,10 @@ class StridedArrayView1D(unittest.TestCase):
self.assertEqual(bytes(b), b'hello') self.assertEqual(bytes(b), b'hello')
self.assertEqual(b.size, (5, )) self.assertEqual(b.size, (5, ))
self.assertEqual(b.stride, (1, )) self.assertEqual(b.stride, (1, ))
# We don't provide typed access for views created from buffers, so the # We get B as "general data", consistently with what memoryview() does
# format is unspecified to convey "generic data" # for bytes/bytearray
self.assertEqual(b.format, None) self.assertEqual(memoryview(a).format, 'B')
self.assertEqual(b.format, 'B')
self.assertEqual(b[2], ord('l')) self.assertEqual(b[2], ord('l'))
self.assertEqual(sys.getrefcount(a), a_refcount + 1) self.assertEqual(sys.getrefcount(a), a_refcount + 1)
@ -296,6 +297,19 @@ class StridedArrayView1D(unittest.TestCase):
del b del b
self.assertTrue(sys.getrefcount(a), a_refcount) self.assertTrue(sys.getrefcount(a), a_refcount)
def test_init_buffer_no_format(self):
a = b'hello'
# ArrayView doesn't preserve the format, so this should then get None
# for the format, instead of B
b = containers.StridedArrayView1D(containers.ArrayView(a))
self.assertEqual(len(b), 5)
self.assertEqual(bytes(b), b'hello')
self.assertEqual(b.size, (5, ))
self.assertEqual(b.stride, (1, ))
self.assertEqual(b.format, None)
self.assertEqual(b[2], ord('l'))
def test_init_buffer_empty(self): def test_init_buffer_empty(self):
a = b'' a = b''
a_refcount = sys.getrefcount(a) a_refcount = sys.getrefcount(a)
@ -305,7 +319,7 @@ class StridedArrayView1D(unittest.TestCase):
self.assertEqual(len(b), 0) self.assertEqual(len(b), 0)
self.assertEqual(b.size, (0, )) self.assertEqual(b.size, (0, ))
self.assertEqual(b.stride, (1, )) self.assertEqual(b.stride, (1, ))
self.assertEqual(b.format, None) self.assertEqual(b.format, 'B')
self.assertEqual(sys.getrefcount(a), a_refcount) self.assertEqual(sys.getrefcount(a), a_refcount)
def test_init_buffer_memoryview_obj(self): def test_init_buffer_memoryview_obj(self):
@ -326,9 +340,10 @@ class StridedArrayView1D(unittest.TestCase):
b = containers.MutableStridedArrayView1D(a) b = containers.MutableStridedArrayView1D(a)
self.assertEqual(b.size, (5, )) self.assertEqual(b.size, (5, ))
self.assertEqual(b.stride, (1, )) self.assertEqual(b.stride, (1, ))
# We don't provide typed access for views created from buffers, so the # We get B as "general data", consistently with what memoryview() does
# format is unspecified to convey "generic data" # for bytes/bytearray
self.assertEqual(b.format, None) self.assertEqual(memoryview(a).format, 'B')
self.assertEqual(b.format, 'B')
self.assertEqual(b[4], ord('?')) self.assertEqual(b[4], ord('?'))
b[4] = ord('!') b[4] = ord('!')
self.assertEqual(b[4], ord('!')) self.assertEqual(b[4], ord('!'))
@ -1033,6 +1048,10 @@ class StridedArrayView4D(unittest.TestCase):
containers.StridedArrayView4D().transposed(4, 3) containers.StridedArrayView4D().transposed(4, 3)
class StridedArrayViewCustomType(unittest.TestCase): class StridedArrayViewCustomType(unittest.TestCase):
# This tests exposing statically typed StridedArrayView instances from C++,
# see StridedArrayViewCustomDynamicType below for types specified
# dynamically and types inherited from the buffer protocol
def test_short(self): def test_short(self):
a = test_stridedarrayview.get_containers() a = test_stridedarrayview.get_containers()
self.assertEqual(type(a.view), containers.StridedArrayView2D) self.assertEqual(type(a.view), containers.StridedArrayView2D)
@ -1080,10 +1099,7 @@ class StridedArrayViewCustomType(unittest.TestCase):
# as memoryview can't handle their types # as memoryview can't handle their types
class StridedArrayViewCustomDynamicType(unittest.TestCase): class StridedArrayViewCustomDynamicType(unittest.TestCase):
# TODO test construction from a (typed) array or memory view, should work def test_binding_float(self):
# and now it doesn't
def test_float(self):
a = test_stridedarrayview.MutableContainerDynamicType('f') a = test_stridedarrayview.MutableContainerDynamicType('f')
self.assertEqual(a.view.size, (2, 3)) self.assertEqual(a.view.size, (2, 3))
self.assertEqual(a.view.stride, (12, 4)) self.assertEqual(a.view.stride, (12, 4))
@ -1097,7 +1113,7 @@ class StridedArrayViewCustomDynamicType(unittest.TestCase):
self.assertEqual(a.view[1][1], 0.0) self.assertEqual(a.view[1][1], 0.0)
self.assertEqual(a.view[1][2], 0.0) self.assertEqual(a.view[1][2], 0.0)
def test_int(self): def test_binding_int(self):
a = test_stridedarrayview.MutableContainerDynamicType('i') a = test_stridedarrayview.MutableContainerDynamicType('i')
self.assertEqual(a.view.size, (2, 3)) self.assertEqual(a.view.size, (2, 3))
self.assertEqual(a.view.stride, (12, 4)) self.assertEqual(a.view.stride, (12, 4))
@ -1111,6 +1127,52 @@ class StridedArrayViewCustomDynamicType(unittest.TestCase):
self.assertEqual(a.view[1][1], -773) self.assertEqual(a.view[1][1], -773)
self.assertEqual(a.view[1][2], 0) self.assertEqual(a.view[1][2], 0)
def test_init_float(self):
a = array.array('f', [1.5, 0.75, 103.125])
b = containers.MutableStridedArrayView1D(a)
self.assertEqual(b.size, (3,))
self.assertEqual(b.stride, (4,))
self.assertEqual(b.format, 'f')
b[1] *= 3.0
self.assertEqual(b[0], 1.5)
self.assertEqual(b[1], 2.25)
self.assertEqual(b[2], 103.125)
def test_init_short(self):
a = array.array('H', [12, 247, 65535, 2206])
b = containers.MutableStridedArrayView1D(a)
self.assertEqual(b.size, (4,))
self.assertEqual(b.stride, (2,))
self.assertEqual(b.format, 'H')
b[2] -= 12765
self.assertEqual(b[0], 12)
self.assertEqual(b[1], 247)
self.assertEqual(b[2], 52770)
self.assertEqual(b[3], 2206)
def test_init_strided(self):
a = array.array('f', [2.5, 3.14, -1.75, 53.2])
b = memoryview(a)[::2]
c = containers.StridedArrayView1D(b)
self.assertIs(c.owner, b)
self.assertEqual(c.size, (2,))
self.assertEqual(c.stride, (8,))
self.assertEqual(c.format, 'f')
self.assertEqual(c[0], 2.5)
self.assertEqual(c[1], -1.75)
def test_init_access_not_implemented(self):
# TODO numpy np.int64 results in l even though python's struct
# classifies that as a 4-byte type, what the fuck?!
a = array.array('L', [1, 2, 3])
b = containers.MutableStridedArrayView1D(a)
self.assertEqual(b.format, 'L')
with self.assertRaisesRegex(NotImplementedError, "access to this data format is not implemented, sorry"):
b[2]
with self.assertRaisesRegex(NotImplementedError, "access to this data format is not implemented, sorry"):
b[1] = 5
class BitArray(unittest.TestCase): class BitArray(unittest.TestCase):
def test_init(self): def test_init(self):
a = containers.BitArray() a = containers.BitArray()

34
src/python/corrade/test/test_containers_numpy.py

@ -34,8 +34,11 @@ except ModuleNotFoundError:
raise unittest.SkipTest("numpy not installed") raise unittest.SkipTest("numpy not installed")
class StridedArrayViewCustomType(unittest.TestCase): class StridedArrayViewCustomType(unittest.TestCase):
# short and mutable_int tested in test_containers, as for those memoryview # This tests exposing statically typed StridedArrayView instances from C++,
# works well... well, for one dimension it does # see StridedArrayViewCustomDynamicType below for types specified
# dynamically and types inherited from the buffer protocol. The short and
# mutable_int variants tested in test_containers, as for those memoryview
# works well... well, for one dimension it does.
def test_mutable_vector3d(self): def test_mutable_vector3d(self):
a = test_stridedarrayview.MutableContainer3d() a = test_stridedarrayview.MutableContainer3d()
@ -132,7 +135,7 @@ class StridedArrayViewCustomType(unittest.TestCase):
]) ])
class StridedArrayViewCustomDynamicType(unittest.TestCase): class StridedArrayViewCustomDynamicType(unittest.TestCase):
def test_short_short(self): def test_binding_short_short(self):
a = test_stridedarrayview.MutableContainerDynamicType('hh') a = test_stridedarrayview.MutableContainerDynamicType('hh')
self.assertEqual(a.view.size, (2, 3)) self.assertEqual(a.view.size, (2, 3))
self.assertEqual(a.view.stride, (12, 4)) self.assertEqual(a.view.stride, (12, 4))
@ -155,3 +158,28 @@ class StridedArrayViewCustomDynamicType(unittest.TestCase):
self.assertEqual(a.view[1][0], (22563, -17665)) self.assertEqual(a.view[1][0], (22563, -17665))
self.assertEqual(a.view[1][1], (-22, 18)) self.assertEqual(a.view[1][1], (-22, 18))
self.assertEqual(a.view[1][2], (0, 0)) self.assertEqual(a.view[1][2], (0, 0))
def test_init_long(self):
a = np.array([[1, 2, 3], [-4, 5000000000, 6]], np.dtype('q'))
self.assertEqual(a.dtype, 'int64')
b = containers.MutableStridedArrayView2D(a)
self.assertEqual(b.size, (2, 3))
self.assertEqual(b.stride, (24, 8))
self.assertEqual(b.format, 'q')
b[1, 1] *= 2
self.assertEqual(b[0, 2], 3)
self.assertEqual(b[1, 0], -4)
self.assertEqual(b[1, 1], 10000000000)
def test_init_double(self):
a = np.array([[[1.0], [2.0]], [[-4.0], [5.0]]], np.dtype('d'))
self.assertEqual(a.dtype, 'float64')
b = containers.MutableStridedArrayView3D(a)
self.assertEqual(b.size, (2, 2, 1))
self.assertEqual(b.stride, (16, 8, 8))
self.assertEqual(b.format, 'd')
b[1, 1, 0] *= -2.0
self.assertEqual(b[0, 1, 0], 2.0)
self.assertEqual(b[1, 1, 0], -10.0)

Loading…
Cancel
Save