DigitalWatch 2.0.0.7 to 2.0.0.*, And then there was scanning Options

[nate]

DW 2.0.0.7 200K
Source Code - 2.0.0.7 tag
Latest Changes - trunk

Note: This version updates the channels.xml file so if you want to keep your old one make sure you make a copy of it.
For this version i'd like people to start without a channels file at all to test out the scanning.

Changes

• Implemented Channel Scanning. This required heaps of other changes.
  • Renamed "Program" in DVBTChannels to "Service".
  • Added a command queue thread to TVControl to allow UpdateChannel events to be handled.
    This also allows the OSD to be drawn while changing channels.
  • Reworked BDADVBTSource to be able to tune to frequencies without channels entries so that scanning can be done for that frequency.
• Removed all state information (such as current network) from DVBTChannels. It's the wrong place for it.
• Added menu options for adding and ordering networks.
• Added FrequencyList to show a list of frequencies (for Australia) when adding networks.
• Added ability for an OSDList to contain both items and itemLists.
• Fixed DirectDraw OSD so the overlay color blt rectangle doesn't go out of bounds.
• Commented out the FPS display.
• Added DScaler video decoder
• Fixed an exception when application is closing. There may still be a problem though.
• Fixed stuttering by setting the reference clock to the DW demux instead of the default demux.

Due to the change from using indexes to actual transport stream and service ID's SetChannel(x) now needs a transport stream id, so the default keys 1-9 will not work out of the box. Once you've done a channel scan you can check the channels.xml file for the tsid and put that in keys.xml to fix it. I plan to implement a favorites system to get around having to do this. 1-9 will choose favorites which can be set to any network or service from within the menus.

When adding a network there's no indication if scanning fails except that the scanning text will disappear. If scanning succeeds then the first service detected should start playing. Once this starts the scanning text will reappear but you don't have to wait for it. You can select the next network as soon as you want. It's just rescanning since the graph was stopped and started when the first service was selected.

There's no signal strength or quality displayed yet.

Use F1 and F2 to bring up the main and tv menus.

An unfortunate side effect of the changes i made in the channels structure to support scanning is that it doesn't remember the last service from each network anymore. I will fix this so don't worry.

I havn't finished creating all the tv menu pages yet. I will do a service setup menu similar to the TV Channel menu that will allow you to reorder services the same way networks can be reordered. In the meantime you can just do it manually in the channels.xml file.

If you're not in Australia then feel free to edit the FrequencyList.xml file to your needs.

I'm sure there's more stuff I need to mention but that'll do for now.

Check the DigitalWatch.log and BDA_DVB-T\Scanning.log files if find any problems.

[bear]

Hi nate,

For this version i'd like people to start without a channels file at all to test out the scanning.

Manual scanning worked first time for me. Very nice & very quick too.

[PhilbyJ5]

nate,

Ahhhh...Channel Scanning. Terrific work.

Scanning went fine for most channels but crashed when trying to scan SBS in Canberra. Have attached a screen shot and log file.

Attached File(s)

 DigitalWatch.txt ( 16.27K ) Number of downloads: 44
 DW2007_Scanning_Crash.jpg ( 190.54K ) Number of downloads: 165

 

[nate]

Hi Philby,

The DigitalWatch.log file you posted isn't the one from straight after the crash. It looks like you've run DW again afterward, and each time you restart DW the log file is cleared. I did spot this problem though.

CODE

Failed to load filter: DScaler Video Decoder  error number -2147221164  The filter is not registered.

You don't have the DScaler video decoder.
I changed the MediaTypes.xml file to use the DScaler video decoder during testing and didn't change it back.

Just edit your MediaTypes.xml file and change the decoder for MPEG2 Video to one that you have. See the Decoders.xml file for a list of decoders.
The previous versions of DW2 were using <Decoder name="Mpeg2Dec Video Decoder" />

Is the scanning crash repeatable? Delete your channels.xml file and try scanning again and grab a copy of DigitalWatch.log and Scanning.log straight after the crash.

Cheers,
Nate

[PhilbyJ5]

Hi nate,

Doh. Sorry about posting the wrong log.
I did spot the decoder error in the log but didn't think it would be related to the scanning error. I just hadn't quite gotten to the decoder setup yet. Will finish the setup now and report back.

[PhilbyJ5]

Deleted logs (left the decoder as DScaler to try one thing at a time) and tried channel scanning again and it's all good. I couldn't reproduce the earlier crash. Scanning was fast and worked for all channels including SBS. Do you want the logs from the successful scans? Are they of any use?

Changed to the Cyberlink Power DVD decoders with VMR 9 and video decoding now shows the OSD. Nice one.

[nate]

Deleted logs (left the decoder as DScaler to try one thing at a time)

Good thinking.

and tried channel scanning again and it's all good.  I couldn't reproduce the earlier crash.  Scanning was fast and worked for all channels including SBS.  Do you want the logs from the successful scans?  Are they of any use?

No. I doubt i'd find anything that would help me figure out what caused the crash so i won't waste time on it.

Changed to the Cyberlink Power DVD decoders with VMR 9 and video decoding now shows the OSD.  Nice one.

Well that's impressive. You're using a feature i havn't implemented yet. At the moment you should only get the OSD when using the overlay mixer, or if video isn't playing.

[Guest_DigiWatchSaver_*]

For this version i'd like people to start without a channels file at all to test out the scanning.

Due to the change from using indexes to actual transport stream and service ID's SetChannel(x) now needs a transport stream id, so the default keys 1-9 will not work out of the box. Once you've done a channel scan you can check the channels.xml file for the tsid and put that in keys.xml to fix it. I plan to implement a favorites system to get around having to do this. 1-9 will choose favorites which can be set to any network or service from within the menus.

Hi Nate,

I've been trying your latest version and found a few issues with the channels scanning and DW itself

First of all the scanning, In Sydney, when you scan for Nine (191500) and D44 (578500), you have something a bit strange happening, instead of having 2 separate listing, DW combine both under one network

<Network Frequency="578500" Bandwidth="7" TransportStreamId="1" Name="DIGITAL FORTY FOUR">
<Service serviceId="1" name="NINE DIGITAL" LCN="9">
<Stream PID="519" Active="1" Type="Video" />
<Stream PID="720" Active="1" Type="MPEG-2 Audio" />
<Stream PID="721" Active="1" Type="AC3 Audio" />
<Stream PID="583" Active="1" Type="Teletext" />
<Stream PID="2306" Active="1" Type="Unknown" />
<Stream PID="2307" Active="1" Type="Unknown" />
<Stream PID="2308" Active="1" Type="Unknown" />
</Service>
<Service serviceId="5" name="NINE HD" LCN="90">
<Stream PID="512" Active="1" Type="Video" />
<Stream PID="650" Active="1" Type="AC3 Audio" />
<Stream PID="576" Active="1" Type="Teletext" />
</Service>
<Service serviceId="6" name="NINE GUIDE" LCN="99">
<Stream PID="517" Active="1" Type="Video" />
<Stream PID="700" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="4" name="FTVguide" LCN="4">
<Stream PID="150" Active="1" Type="Video" />
<Stream PID="151" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="40" name="D44" LCN="40">
<Stream PID="100" Active="1" Type="Video" />
</Service>
<Service serviceId="41" name="D44-News" LCN="41">
<Stream PID="110" Active="1" Type="Video" />
</Service>
<Service serviceId="44" name="D-44" LCN="44">
<Stream PID="100" Active="1" Type="Video" />
</Service>
<Service serviceId="45" name="D44-NSW" LCN="45">
<Stream PID="120" Active="1" Type="Video" />
<Stream PID="121" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="46" name="D44-ACC" LCN="46">
<Stream PID="170" Active="1" Type="Video" />
<Stream PID="171" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="47" name="D44-Macquarie" LCN="47">
<Stream PID="160" Active="1" Type="Video" />
<Stream PID="161" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="48" name="D44" LCN="48">
<Stream PID="100" Active="1" Type="Video" />
</Service>
<Service serviceId="49" name="D44-EXPO" LCN="49">
<Stream PID="140" Active="1" Type="Video" />
<Stream PID="141" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="401" name="D44Parl-Reps" LCN="145">
<Stream PID="103" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="402" name="D44Parl-Senate" LCN="146">
<Stream PID="104" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="403" name="D44Parl-ComReps" LCN="147">
<Stream PID="105" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="404" name="D44Parl-1Com'tee" LCN="148">
<Stream PID="106" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="405" name="D44Parl-2Com'tee" LCN="149">
<Stream PID="107" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="406" name="D44Parl-3Com'tee" LCN="150">
<Stream PID="108" Active="1" Type="MPEG-2 Audio" />
</Service>
<Service serviceId="407" name="D44Parl-4Com'tee" LCN="151">
<Stream PID="113" Active="1" Type="MPEG-2 Audio" />
</Service>
</Network>

From what I understand, the issue is comming from the TransportStreamId="1" for both Nine and D44.

If you split the 2 channels manually, when you try to access D44 channels, you have the channels for Nine listed and can't see anything from D44.

Another issue is that DW start to allocate memory at a quite fast rate and doesn't stop allocating. I've let it go up to 300MB and then stopped as my computer was becoming pretty unstable. This problem seems to occur as soon as DW start and even before I select the watch TV or Channel Setup. CPU usage of DW is arround 30% even when doing nothing, I mean when I just select Add Network and then don't select anything and wait, the memory increase and the CPU usage is between 25 and 40%

Also if you re-run DW then go in watch TV and watch you fist network
then you press F1 and select Watch TV
there you hit the enter key and you will see that the list of channels has the list twice if you do it again the list of channels get added again and again...

I also had DW to comletely freeze during scanning procedure but can't figure out why.

Cheers

[edit]
Oh I forgot also I've noticed that when the Network Names are displayed, some of them don't fit in the Bitmap button, for example Nine Network Australia is cut and you can't see the ia of Australia. I have the same issue with DIGITAL FOURTY FOUR where the last UR is not displayed
[/edit]

[Guest_NoneToBe_*]

Nate,

I tried this and the channel scanning bit seems to work for updating a channels.xml file but I've got an Nvida Fx5600 and it doesn't seem to be displaying any video - here is an extract from the log:

CODE

       Finished Initialising BDA Source
       Playing BDA Source
         Building Graph (177500, 7)
           Stopping DW Graph
           Finished Stopping DW Graph
           Unloading Tuner
           Finished Unloading Tuner
           Cleaning up DW Graph
           Finished Cleaning up DW Graph
           Loading Tuner
           Finished Loading Tuner
           Adding Demux Pins
             Creating pin: PID=769   Name="Video"
               Rendering stream of type "MPEG2 Video" with decoder "Intervideo Video Decoder - VMR9"
             Creating pin: PID=772   Name="Teletext"
             Creating pin: PID=770   Name="Audio"
               Rendering stream of type "MPEG Audio" with decoder "MPEG Audio Decoder"
           Finished Adding Demux Pins
           Starting DW Graph
             Reference Clock is "DW MPEG-2 Demultiplexer"
           Finished Starting DW Graph
         Finished Setting Channel
TVControl::ExecuteGlobalCommand - Key(88, 0, 0, 0)
 TVControl::Key -   X (88)
   TVControl::ExecuteGlobalCommand - Exit()
Destroying BDA Source
 Stopping DW Graph
 Finished Stopping DW Graph
 Unloading Tuner
 Finished Unloading Tuner
 Cleaning up DW Graph
 Finished Cleaning up DW Graph
Finished Destroying BDA Source
Destroying BDA Source
Finished Destroying BDA Source

On the positive side, it seems to automagically pick up the BDA driver for the new PocketHDTV USB2.0 device and a reorder of cards.xml makes it use it but again no video

DW701 in another directory still displays video fine but doesn't work with my new USB2.0 toy (obviously).

[PhilbyJ5]

Nate,

(PhilbyJ5 @ Jul 11 2005, 05:05 PM)
Changed to the Cyberlink Power DVD decoders with VMR 9 and video decoding now shows the OSD.  Nice one.

Well that's impressive. You're using a feature i havn't implemented yet. At the moment you should only get the OSD when using the overlay mixer, or if video isn't playing.

Fair call. Just checked again and sure enough, I only get the OSD on first load (ie choose Watch TV) and then the OSD with channel and network appears just before the video kicks in. You're spot on, after the video is rendered, theres no OSD. My bad.

[nate]

First of all the scanning, In Sydney, when you scan for Nine (191500) and D44 (578500), you have something a bit strange happening, instead of having 2 separate listing, DW combine both under one network
...
From what I understand, the issue is comming from the TransportStreamId="1" for both Nine and D44.

Serves me right for assuming that broadcasters would be smart enough to have unique transport stream ids. From memory, in melbourne ABC is two hundred and something, SBS is three hundred and something, 7 is four hundred and something, 9 is five hundred and something and 10 is six hundred and something. So i figured that certain id ranges had been assigned to each network. A transport stream id of 1 seems very odd.

Ok, so what am i going to use to uniquely identify network. Can you check Scanning.log and see if the network id is unique?

Another issue is that DW start to allocate memory at a quite fast rate and doesn't stop allocating. I've let it go up to 300MB and then stopped as my computer was becoming pretty unstable. This problem seems to occur as soon as DW start and even before I select the watch TV or Channel Setup.

At the first menu i see about an 8k/s memory leak that stops if i press escape, so it looks like theres a leak in the osd.
I ran tv for 10 minutes this morning making sure no osd was visible and didn't notice much in the way of a memory increase so hopefully it's just the osd.

CPU usage of DW is arround 30% even when doing nothing, I mean when I just select Add Network and then don't select anything and wait, the memory increase and the CPU usage is between 25 and 40%

The cpu usage will vary a little depending on the directdraw capabilities of your video card. On my machine there's a 20%-30% more cpu used when the add network menu is visible. That menu should be the most cpu intesive one since it's doing the most directdraw blits.

Also if you re-run DW then go in watch TV and watch you fist network
then you press F1 and select Watch TV
there you hit the enter key and you will see that the list of channels has the list twice if you do it again the list of channels get added again and again...

I'll look into this. I'm probably not clearing the channel list when i stop and restart the BDA source.

I also had DW to comletely freeze during scanning procedure but can't figure out why.

I had this happen once in testing a while ago but never figured out why. When it happened for me it seemed that the tables and sections filter stopped raising events so it might only happen if something weird is detected in the transport stream.

[edit]  Oh I forgot also I've noticed that when the Network Names are displayed, some of them don't fit in the Bitmap button, for example Nine Network Australia is cut and you can't see the ia of Australia. I have the same issue with DIGITAL FOURTY FOUR where the last UR is not displayed
[/edit]

That's easy to fix. Just make the width a bit bigger or the font a bit smaller in the OSD.xml file.

CODE

<window name="NetworkList">
...
<list name="Networks">
...
<size width=250 height=400 />
...
<font height=30 weight=700 color="#FFFFFF" />

I tried this and the channel scanning bit seems to work for updating a channels.xml file but I've got an Nvida Fx5600 and it doesn't seem to be displaying any video - here is an extract from the log:

Does it work using a different decoder and the overlay mixer. Aside from disabling the osd i havn't done anything to support VMR9 rendering yet.

On the positive side, it seems to automagically pick up the BDA driver for the new PocketHDTV USB2.0 device and a reorder of cards.xml makes it use it but again no video

automagically? Hahaha I assure you that nothing in DW is magic. It's was all done through long hours, lots of trial and error, and the occasional prayer.

[Guest_DigiWatchSaver_*]

Serves me right for assuming that broadcasters would be smart enough to have unique transport stream ids. From memory, in melbourne ABC is two hundred and something, SBS is three hundred and something, 7 is four hundred and something, 9 is five hundred and something and 10 is six hundred and something. So i figured that certain id ranges had been assigned to each network. A transport stream id of 1 seems very odd.

Ok, so what am i going to use to uniquely identify network. Can you check Scanning.log and see if the network id is unique?

I checked and the network id seems different. I've rescanned all Sydney channels and attached both channels.xml and Scanning.log. everything is as is, by that I mean that I haven't split Nine and D44 so you can see exactly what I get. [attachment=191:attachment]

At the first menu i see about an 8k/s memory leak that stops if i press escape, so it looks like theres a leak in the osd.
I ran tv for 10 minutes this morning making sure no osd was visible and didn't notice much in the way of a memory increase so hopefully it's just the osd.

Yep, same here if the menu is not displayed, I have no memory loss...

The cpu usage will vary a little depending on the directdraw capabilities of your video card. On my machine there's a 20%-30% more cpu used when the add network menu is visible. That menu should be the most cpu intesive one since it's doing the most directdraw blits.

Ok I thought the menu was just a bitmap displayed and didn't realize you were using directdraw for that, that's why I was surprised to see DW using that much cpu to display bitmaps

I had this happen once in testing a while ago but never figured out why. When it happened for me it seemed that the tables and sections filter stopped raising events so it might only happen if something weird is detected in the transport stream.

This happened once or twice here and always when scanning and a truck/bus or an helicopter is passing by which might explain a stream corruption... Maybe is there a way to put a fail-safe there to prevent that?

That's easy to fix. Just make the width a bit bigger or the font a bit smaller in the OSD.xml file.

CODE

<window name="NetworkList">
...
<list name="Networks">
...
<size width=250 height=400 />
...
<font height=30 weight=700 color="#FFFFFF" />

Nice, this one was pretty easy

Thanks for all that.

If you need anything else, please let me know

[nate]

New Version - DW 2.0.0.9

Changes

• Fixed channels file for when more than one broadcaster used the same transport stream id.
• Fixed problem where selecting Watch TV a second time would duplicate all the channels.
• Fixed problem in scanning. The waiting filters weren't cleared if there was a timeout scanning a table.
• Fixed threading issue with data items and lists.
• Fixed memory leak in OSD code

I've changed the scanning and channels code to create networks based on their transport stream id, original network id, and network id, instead of just the transport stream id. That should stop services from one network getting mixed up with another.

The main identifier for networks is now the original network id instead of the transport stream id. That means if you want to use SetChannel() you need to supply the ONID value the same as in the channels file.

I've changed the channels.xml file to store IDs in hex instead of decimal because i got sick of converting between the values in the scanning log and the channels file.

[nate]

I had this happen once in testing a while ago but never figured out why. When it happened for me it seemed that the tables and sections filter stopped raising events so it might only happen if something weird is detected in the transport stream.

This happened once or twice here and always when scanning and a truck/bus or an helicopter is passing by which might explain a stream corruption... Maybe is there a way to put a fail-safe there to prevent that?

The problem is, having only had it happen once, it's very difficult to tell exactly where the problem is, so i wouldn't know where to begin to put fail-safe's in. If someone can reproduce this regularly and provide me with log files right after the lockup then it might help.

[renura]

Hi Nate,

Downloaded it and within minutes had scannel all channles and was wathing TV, good work,

If you have a number of different devices in your PC, is there an easy way of telling it which one to use?

Cheers

[nate]

Downloaded it and within minutes had scannel all channles and was wathing TV, good work,

Good to hear

If you have a number of different devices in your PC, is there an easy way of telling it which one to use?

In the BDA_DVB-T\Cards.xml file just set active to 0 for the cards you don't want to use.

[renura]

Downloaded it and within minutes had scannel all channles and was wathing TV, good work,

Good to hear

If you have a number of different devices in your PC, is there an easy way of telling it which one to use?

In the BDA_DVB-T\Cards.xml file just set active to 0 for the cards you don't want to use.

Thanks

[Guest_DigiWatchSaver_*]

• Fixed channels file for when more than one broadcaster used the same transport stream id.

I tried to scan all network availlable in Sydney and the result is still not god this time instead of having only Nine and D44 together, I have Nine, SBS and D44. Please see first attachement for both Channels.xml and Scanning.log [attachment=198:attachment]
Why can't you just use the frequency or an index based on insertion (like first network inserted and that can be locked would be index one and so forth) to uniquely identify networks? This might be safer don't you think?

• Fixed memory leak in OSD code

I don't know if it's coming from OSD or not but I still have 2 cases of Memory leak.
First when I scan, as soon as I enter the Add Network Menu, the memory allocated increase at a quite high speed (about 400-500k per second)
the second is when I watch something, as soon as I select Watch TV Menu. This one is a bit more trickier I think as the memory goes up then down then up then down... resulting in a very slow increase I've attached the DigitalWatch.log [attachment=199:attachment] just in case...

Keep up the good work Nate!

[nate]

I tried to scan all network availlable in Sydney and the result is still not god this time instead of having only Nine and D44 together, I have Nine, SBS and D44. Please see first attachement for both Channels.xml and Scanning.log

Thanks. I've forgot about some places where it references networks by transport stream id. I'm not sure why SBS is in there too, but i think i can fix it.

Why can't you just use the frequency or an index based on insertion (like first network inserted and that can be locked would be index one and so forth) to uniquely identify networks? This might be safer don't you think?

I thought about using the frequency, but the frequency in the transport stream doesn't always match the frequency you're actually tuned too.
I was originally using indexes, but it made dynamic updates really difficult. I guess it was more of an issue with services than networks, but if i can get this working it should be great. If i was still using indexes it would be very hard to reorder the networks on the fly.

I'm more curious as to why there are so many timeouts scanning D44.

I don't know if it's coming from OSD or not but I still have 2 cases of Memory leak.
First when I scan, as soon as I enter the Add Network Menu, the memory allocated increase at a quite high speed (about 400-500k per second)

I'll check it out.

The second is when I watch something, as soon as I select Watch TV Menu. This one is a bit more trickier I think as the memory goes up then down then up then down... resulting in a very slow increase

This one's trick. It's hard to distinguish between a memory leak and the scanning code building up it's data structures. If it's still going up after a few minutes then it's probably a memory leak.

[Guest_DigiWatchSaver_*]

Thanks. I've forgot about some places where it references networks by transport stream id. I'm not sure why SBS is in there too, but i think i can fix it.

No problems, that's the least we can do with all the work YOU do...

I'm more curious as to why there are so many timeouts scanning D44.

Well, the thing is I'm using rabbit hears to watch DTV and live close to a main road so each time a truck or a bus pass in front of my house the signal gets corrupted. During the day that would be about every 10 seconds. This has a different impact depending on which channel I'm watching/scanning. D44 already has a bad signal arround here so combined with the traffic interference this must result in the timeouts when scanning... I'll try and see if I can get something better this week-end as there's usually less traffic...

This one's trick. It's hard to distinguish between a memory leak and the scanning code building up it's data structures. If it's still going up after a few minutes then it's probably a memory leak.

I've let it play here for several minutes and it never stopped going up. Stopped DW when it reached 20MB+

Cheers