LoadTexture asynchronous

May 13, 2013 at 12:42 AM
Do you support methods for loading textures or other resources async?
May 13, 2013 at 3:55 AM
You could potentially add that functionality to the renderer...sort of.Make a ResourceLoadTask that inherits Task and have a string with the resource name and a flag with the type set in it and then add the load code in the Execute method, so then you just set up the task, add it to the queue and it gets executed by the renderer.AFAIK calling the device to create resources from multiple threads is safe, or maybe there was some flag you need to set on device creation :?
Coordinator
May 14, 2013 at 2:56 AM
That is actually a great question. I currently don't have asynchronous texture loading, but it could be a very compelling feature to add. Currently I rely on the DirectXTK framework to handle all of the texture loading and texture saving (for screen shots). However, I have to include the entire framework for only two or three method calls, which is not so great.

I have been thinking of eliminating the framework and just providing a few simple loaders, just to reduce the amount of maintenance required for the DXTK library. I also had not thought to use multithreaded loading, but that would probably even make a great demo program.

Regarding the Task structure, I think that is an appropriate way to do it, and that is likely how I will implement it. You don't need any special flags on device creation (you actually have to add a special flag to go single threaded, rather than the other way around) - it assumes multithreaded support in the standard configuration. And it is also true that you can call the device methods for creating the texture from multiple threads. However, after loading the texture, then creating the resource to hold it, we then have to update the renderer's data structures to hold the reference to the new texture.

This will require synchronization, since the STL is not thread safe for reading while writing. But even so, if we utilize some type of loading period during the initialization phase, then we could create and load the resources, and then process the loaded data sequentially afterwards. That could work, and would be an interesting use case. In addition, this concept should be extendable to a streaming concept too. Let me think about that one for a little bit to work through if any other issues might come about...

Great suggestions guys - I think this is going into the engine!
May 14, 2013 at 5:34 AM
Great that you like the suggestion. I think its a "need need need" improvement.
Editor
May 14, 2013 at 5:59 AM
Hi,

I was wondering, how are you using this and what is the "need"? Is it to reduce initial startup time or are you thinking about say, megatexture streaming or something else? I ask this because I've been grappling with this as well and am genuinely curious about specific use-cases and where it comes in handy.

Most of the time I need to load the required texture on startup anyway since I can't wait to load it when I need it, and it's not clear to me how to "predict" I'll need a texture unless I have very specific points like level or scene loading. And another case -- spawning lots of async reads on initial loading of a "level" -- seems definitely not a win because it just thrashes the disk more and actually slows everything down. So for that case I end up with one-by-one background loading - sort of async, but a pretty trivial version of it. The only other case I have is for very specific situations like say terrain normal maps where I'm generating them on the fly and can get rendering without them until their ready. So just wondering: what are you planning on using such a capability for?

Thanks,
Jason


May 18, 2013 at 12:11 AM
I think he is referring to the usage of an async loader for megatexture streaming, however it would probably first be a good idea to switch to a x64 configuration to allow massive use of RAM to store most recently texture chunks.
Coordinator
May 18, 2013 at 1:43 PM
@TheOutsourcer: I think you are right - he was talking about doing a megatexture implementation before, so it is likely to do with streaming. It is interesting that you mentioned switching to an x64 build, because I was actually thinking of reworking the library projects a little bit. This would make separate build output SDK folders for each build configuration and platform.

I want to be able to target Windows Store Apps and WP8 at some point in the future, so I need to work those build configurations into the mix as well. That will probably be coming sooner rather than later...
Jul 19, 2013 at 3:50 AM
I want to bring up this thread again. I know that your time is limited, but if you have someday time async loading of resources is such a need need need ^^.
Coordinator
Jul 19, 2013 at 12:24 PM
The chances of getting something like this are improving :) The new tiled resources in D3D11.2 seem to be a good fit for such a method, and would make a nice sample program. It is likely that I will build something like this, but as you mentioned time is a constraint at the moment...

Is there something I can help you with prior to building a whole new sample? Need ideas or suggestions? Have requirements I should take into consideration? Two brains are better than one, especially when talking about multithreading requirements!
Jul 19, 2013 at 3:59 PM
Edited Jul 19, 2013 at 4:22 PM
Ive heard about the new feature of D3D11.2 ... but its not Windows7 or Windows8 .. i heard its only Windows8.1
But this offers a good way to realize a MegaTexture (Virtual Texturing) technique.

At the moment i want do something like the DirectX Sample "Content Streaming" ... perhaps you know it. Here is an image of it i found on the web. I need this for having a constant memory in gpu for loading huge textures as terrain patches.

Independet of this async loading of resources is needed when i added a new GameObject to the scene. The game stucks. Sure i can manage that i preload every resource but a async loading feature would be awesome.

Image
Jul 20, 2013 at 8:16 PM
Unfortunately, tiled resources(the D3D11.2 feature you are talking about) will only run on the latest GPUs.I wonder if they'll add a software-emulated version of it.
Coordinator
Jul 29, 2013 at 1:51 AM
I was thinking about how to implement this sample, and it occurred to me that the KinectPlayground sample already implements something along the lines of what you are looking for. Have you taken a look at it yet?

Basically there is a separate thread watching to see when a new Kinect frame is ready to receive, and when it is then the data is copied into a CPU buffer. Once there is data available in that buffer, then on the next frame rendering there is a SceneRenderTask that copies the data into GPU resources, and subsequently uses them.

I think this models what you are looking for quite well, and all you would have to do is replace the KinectManager with a suitable texture data supplier. What do you think about that?