[core][aDAG] suppress numpy is_writable warning

[rkooo567]

Why are these changes needed?

When we do cpu tensor -> gpu transfer, we change tensor to numpy -> and numpy -> tensor to utilize ray's zero copy serialization/deserialization. However, when it happens, ray sets nd_array.is_writable = False.

This is intentional design to make object immutable. Ray object being immutable is important for data integrity. However, when we convert such numpy array to tensor, tensor raises a warning loudly;

(A pid=553715) /home/ubuntu/sang-dev/ray/python/ray/experimental/channel/serialization_context.py:116: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:206.)

which means that tensor doesn't respect is_writable.

However, we use this mechanism (converting zero copy numpy array inside shared memory that has is_writable=False) only when we convert numpy array to gpu object. We know it is safe to do that because the cpu data is anyway copied to gpu (so no one can write to shared memory anyway). Since we know it is safe, there's no meaning of this warning, but confusion. So we suppress it. To avoid having high overhead suppress warning, we use a global variable to make checking as cheap as possible.

Q: Why not just setting flags.is_writable = True? I tried, but it somehow didn't work. I assume there's some kind of mechanism to block this in Ray?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(