How can I display the acquired data on the screen while logging it into the hard disk?

[foolooo]

Hello every one,
 
I'm using LV 8.5 under windows XP OS, and my DAQ board is PCIe-6251 m-series board. I'm doing data acquisition at 1M/s, and after I stop acquiring, the data stored in the buffer will be writtien into the hard disk. Now I also want to display the input data (which are images of a high-speed camera) while acquiring, can I imeplement that? I don't want to slow down the acquisition rate.
I thought maybe I can use 2 queues to store the data, and write 1 queue into a file, and display the other queue. But the display would cost very fewer frames. If one frame contains 4096 pixels (4096 data), then can I skip in the queue so that it displays continuous 4096 pixels every 10K data?
Just to make it clear, assume the data in the queue is like oooooooooooooooooooooooo..., the elements I want to display are like xxxxooooooooxxxxoooooooo... in which 'x' is the element I want to display. Can I implement that in the queue? Or is there some other manner I can display without affecting the writting? Thank you.
 
Regards,
Bo

[Kevin Price]

No promises here, just some thoughts.
At 1 M/sec you'll need to be a little careful with your memory handling.  If you send your array straight from your DAQmx Read into a queue with no other branches, LabVIEW is smart enough not to copy the 1 M/s of samples.  It lets the queue own the memory pointer.  Then when your File Writing code performs a de-queue, once again LV won't need to do a memory copy of all the array elements.
However, if you want to send any of the data elsewhere such as a live user display, there *will* be some data copying going on.  You'll benefit from a good strategy on this.  Others may provide better or more proven ideas, but meanwhile here's what *I'd* do:
I would let the File Writing code take a subset of the data it pulls from the queue and send it out for use by the User Display code.  I would use a Notifier rather than a Queue for this, as it probably isn't crucial if the live display misses an occasional frame.  This approach allows the data acq to collect and enqueue at the max rate you need.  If this data copying bogs things down occasionally, the Queue just buffers data for a while until the File Writing code can start catching up.  The data acq won't be slowed down.
-Kevin P.

[foolooo]

Hi Kevin,
 
Thank you for your reply. I'm not clear about one thing in your reply. It is a good suggestion to use the notifier to display the samples, but did you mean to use the data 'after' it is written? Or copy the data 'before' it is written? As I understand, you suggest to copy the the subset of the data before it is written into the file, but the 'selecting' will cost time (cpu and ram), is that correct? Thank you again.
 
Best wishes
Bo

[Kevin Price]

I don't know the definitive answer, but I'd start by sending the data *after* it has been written to file.  There may be a way to structure the code to give LV enough hints that it won't ever have to make an actual data copy that way.   From reading other threads, I've come to understand that doing an array subset operation does not necessarily result in data copying. 
So if the file write is done and *then* a subset of the written array is passed to a Notifier, LV may likely realize that the Notifier can now simply take over ownership of the original chunk of memory rather than making a copy of the contents.  I suspect (but am not at all sure) that if you branched the data off to a Notifer *before* the file write, a data copy would need to happen.
Attached is a screenshot to illustrate the kind of hint I have in mind.  Putting the notification inside a single-frame-sequence structure helps tell LV that the write must complete execution before performing the array subset and notification.  That should be enough hint that it can give owenership of the array to the Notifier.
<img src="Loading Image...">
-Kevin P.Message Edited by Kevin Price on 07-18-2008 11:54 AM


notify after write1.png:
http://forums.ni.com/attachments/ni/250/41656/1/notify after write1.png

[tst]

Notifiers always make a data copy at the Send Notification node. The reason is that you can obtain a notifier and then ask to get the latest notification even after it occured, so it needs to exist somewhere. I don't have the specific reference on me, but it should be in one of Stephen's posts on LAVA.
A queue does reuse the buffer because the data can only be dequeued in one place, so you would want to use&nbsp;a queue here.

[tst]

Correction, the copy is done at the Wait primitive (which is essentially the same thing, because the only way you're not going to get a copy is if you don't handle the notification at all). <a href="http://forums.lavag.org/index.php?s=&amp;showtopic=10029&amp;view=findpost&amp;p=41195" target="_blank">Here</a>'s Stephen's post.

[Kevin Price]

Thanks for the correction, tst.&nbsp; Yeah, it *does* make sense -- if you were to loop over Wait On Notification with "ignore previous" = True, you'd *expect* to keep retrieving the same stale data.
I think it'd still be my preferred option though, since the data copy would be confined within the GUI chart update code.&nbsp; If I had decided I could live with lossy notifications, I could very likely also live with a little extra lag in the display.&nbsp;&nbsp; There are other kinds of apps that would be better served by using queues.
-Kevin P.

[foolooo]

Thank Kevin and tst, I'll think about your great suggestion. Your words are kind of hard for me, and it will take some time for me to fully understand it.
&nbsp;
Best wishes
Bo

[foolooo]

Hi Kevin,
I followed your suggestion and used the notification to display the array subset. If the subset will copy the data in any case, then I don't need to add a single-frame sequence for that, is that correct? And another problem is, I want to display some subsets of the array but not just one, for it's a long array and the displaying didn't update&nbsp;fast enough. Do you know how I can do that? Thank you for your reply.
Best wishes
Bo

[foolooo]

Sorry, I edited my post, but it didn't work. The reason the displaying didn't update fast is it happened after the dequeue, so I put it before the queue right after the DAQ. I used a branch of the acquired data, and I'm testing if it slow down the acquisition.
&nbsp;
Cheers,
Bo

[Kevin Price]

I'll try to address some specific&nbsp;questions, but first have some general comments.&nbsp; Step back and carefully prioritize the data handling inside your app.&nbsp; I've been guessing that the top priority is to sustain data acq and file storage at a full 1 Meg/sec.&nbsp; I'm treating that as a "must-have."&nbsp;&nbsp; I'm treating the nearly-live user display as a "nice-to-have" where the goal is an update rate that (A) doesn't bog down the data acq and file storage and (B) is fast enough to provide useful feedback to an operator.
&nbsp;
Now I'm sure I'm not the very top expert around these forums on performance issues but I'm not too shabby.&nbsp; So you may find that some of my performance-enhancing suggestions aren't strictly necessary for your specific app right now, but I'm suggesting you try them anyway as habits that are very likely good ones.
&nbsp;

If the subset will copy the data in any case, then I don't need to add a single-frame sequence for that, is that correct?
No, I'm not sure that's correct.&nbsp;&nbsp;I'm not 100% sure either way, but I know that LV *can* sometimes carry array subsets around without needing to perform data copies.&nbsp; The single-frame sequence helps provide LV a hint that the other branch of that array data *must* be done executing before the array subset is performed.&nbsp; LV may also figure that out on its own, or it may make a copy either way for reasons I'm not prepared to explain.&nbsp; But personally, I'd give LV the hint anyway and hope for the best.
&nbsp;

I want to display some subsets of the array but not just one, for it's a long array and the displaying didn't update&nbsp;fast enough. Do you know how I can do that?
Well, it depends.&nbsp; In general, you&nbsp;would extract multiple subsets and then merge them together.&nbsp; To do it with the best possible performance may be a tricky question.&nbsp; Using "Build Array" is probably *not* the best way.&nbsp; Auto-indexing to the output of a For loop with a constant # of iterations probably *is* good, but your app may need more flexibility than that.
&nbsp;
However, you may simply be trying to jam too much information ("it's a long array") into a gui display element that simply isn't capable of updating very quickly.&nbsp; Or you may be fixed on the idea of displaying more information than an operator needs.
&nbsp;
Here's what I would advocate for pretty much *any* high data-rate app.&nbsp; *DECOUPLE* the gui display updates from the data acq and processing code.&nbsp; Architect the code so that the critical high speed stuff *never* waits for the gui stuff.&nbsp;&nbsp; This is the thought process behind my recommendation of Notifiers.&nbsp;&nbsp; A producer (the file storage code) can write to the Notifier at one rate while a consumer (the gui code) can read from it at a different rate.&nbsp;
&nbsp;

The reason the displaying didn't update fast is it happened after the dequeue, so I put it before the queue right after the DAQ. I used a branch of the acquired data
It sounds like your idea of "displaying fast" is more a measure of lag time than of average data rate.&nbsp; Again, this will go back to those overall priorities.&nbsp; My suggestions were intended to help make the data acq and file storage to run quickly, efficiently, and unimpeded.&nbsp; A side effect of the methods I presented is indeed&nbsp;to increase the lag time from when an event occurs until its presence is visible on the display.&nbsp; Branching off the acquired data before it is sent to be Enqueue'd will almost certainly require data copies inside your most critical loop - the data acq loop.&nbsp; I would *not* do it that way.&nbsp; Again, even if it seems to work good enough for today, it would not be a good habit for apps that demand high performance.
&nbsp;
-Kevin P.
&nbsp;

[foolooo]

Hi Kevin,
&nbsp;
Thank you so much for your advices. All you said is very helpful for me. Then I think I would rather put the display function after the DAQ. Thank you again.
&nbsp;
Best wishes
Bo