EPGScan, Transport Stream Parsing Tool Options

[null_pointer]

I have started a new thread for this tool, the last thread got nuked by me accidentally.

This is a simple MS Windows command line tool to parse, extract and save the Event Information Table (EIT) data from the transport stream. This data contains EPG information that can be used in other applications like Web Scheduler.

Read the readme.txt file in the download zip for more information on running the tool.

Download link
http://sourceforge.net/project/showfiles.php?group_id=211694

With the introduction of the new raw EIT xml output and then using the XSLT template system it is possible to create uniform guide data for all stations in most areas. Read the readme.txt file in the xslt directory of the download zip for more information.

Currently SBS is not transmitting EIT data, I have written a simple SBS web site grabber if anyone is interested.

Change Log

1.0.0.8
- fix extra spaces added to to raw EIT output extended description output
- sort program output in start time

1.0.0.6
- fix a bug for stations that do not set the service EIT available bit
- build a valid table number list per service instead of a global one
- add the custom xml output format to allow XSLT conversion per station to produce consistend data
- fix a few small command line and config file bugs

1.0.0.5
- if the device ID is not specified on the command line with -d or in the configuration file then use the first device found

1.0.0.4
- add better command line parameter handling, use -x <value> instead of order based parameters, see the readme for more info
- add a command line parameter -c for specifying an xml configuration file, this file can contain any parameter that is normally passed in on the command line, command line parameters override the configuration file, see the readme for more info

1.0.0.3
- add a sanity filter for table ID, drop tables above the Last_Table_ID of table 0x50

1.0.0.2
- fix for character encoding, still only does simple 0-127 ascii chars

1.0.0.1
- use filter on GetStreamOfSections to only pass through tables 0x50 to 0x5F for EIT scan
- write EPG data from channel if it exists in the scan even if it does not have its EIT available bit set
- only process tables with the current filed set to 1, we are not interested in upcoming table version

1.0.0.0
- initial release

[drdos7]

Hi,

Nice one. Is the web grabber part of the scanner?

Maybe you should add an ABC web site grabber as well, I've found the ABC/ABC2 EIT EPG isn't much good in the last month or so. I hate to say it, but the 3 commercial stations are pretty good now (EPG wise anyway).

dRdoS7.

[null_pointer]

Nice one. Is the web grabber part of the scanner?

no, the grabber is a stand alone java tool, there is no docs for it at the moment but I will upload it later tonight for people that want to play.

I've found the ABC/ABC2 EIT EPG isn't much good in the last month or so.

is it wrong or just missing data?

I hate to say it, but the 3 commercial stations are pretty good now (EPG wise anyway).

that is good to know.

[drdos7]

Hi,

is it wrong or just missing data?

Yeah, not much data.

Look forward to the grabber.

dRdoS7

[drdos7]

Hi,

Anybody else using this?

I have noticed that ABC is taking longer and longer to scan. Anybody having same?

I'm in Geelong BTW.

I've gone to the grabber for ABC now, as for SBS.

Thanks,

dRdoS7.

[boric]

Hi,

Anybody else using this?

I have noticed that ABC is taking longer and longer to scan. Anybody having same?

I'm in Geelong BTW.

Haven't really noticed ABC being any slower, although if you consider the number of events then yeah it's quite slow. I've scheduled EPGscan to run daily in the background so the time it takes doesn't really matter - particularly considering how damn useful this tool is.

I am finding though, that the ABC data loads oddly into 'DNTV Scheduler Pro', I get an 'empty' program of 0 minutes length between each ABC program. This doesn't happen with 7/9/10 and I can't see any obvious difference in the ABC data to the 7/9/10 data. The data is continuous, i.e. there aren't any time gaps between programs in the XML data.

Slightly OT, but what I am finding frustrating is trying to work out whether the issue is with EPGscan or DNTV Scheduler Pro, and is the best solution to simply buy DVB Web Scheduler Pro? (wish I knew they were different when I bought my tuner!) Particularly as it's difficult to tell what features I lose and/or gain from moving to Web Scheduler Pro. Guess I'll have to install WS Pro on another machine and try it.

I've gone to the grabber for ABC now, as for SBS.

Might have to do the same.

[bear]

I am finding though, that the ABC data loads oddly into 'DNTV Scheduler Pro', I get an 'empty' program of 0 minutes length between each ABC program. This doesn't happen with 7/9/10 and I can't see any obvious difference in the ABC data to the 7/9/10 data. The data is continuous, i.e. there aren't any time gaps between programs in the XML data.

The difference in the epg data between ABC & 7/9/10 is that ABC supply event times with accuracy of seconds where the others are rounded up to the minutes.

Slightly OT, but what I am finding frustrating is trying to work out whether the issue is with EPGscan or DNTV Scheduler Pro, and is the best solution to simply buy DVB Web Scheduler Pro? (wish I knew they were different when I bought my tuner!) Particularly as it's difficult to tell what features I lose and/or gain from moving to Web Scheduler Pro. Guess I'll have to install WS Pro on another machine and try it.

This is a known rounding issue of the pre- commercial version of WS and of the latest DNTV SchedulerPro release. For normal DVB-T capturing there is little loss moving to WS Pro and more to gain. DNTV SchedulerPro features more relate to a wider device type usage.

[boric]

The difference in the epg data between ABC & 7/9/10 is that ABC supply event times with accuracy of seconds where the others are rounded up to the minutes.

OK, I understand, guess I'll have to take a closer look at the raw XML as I didn't notice.

This is a known rounding issue of the pre- commercial version of WS and of the latest DNTV SchedulerPro release. For normal DVB-T capturing there is little loss moving to WS Pro and more to gain. DNTV SchedulerPro features more relate to a wider device type usage.

Thanks, I'll give it a whirl.

[Jeltz]

This bit of software works well for me.
Two notes:
1)EPGSCAN on SBS scan says it has no EIT table available (true!), and then proceeds to delay until timeout waiting for EPG before exiting.
2) Not really EPGSCAN issues but
a) the channel 7 EIT entries often have annoying codes in fromt of them like M- for movie etc. These seem to be station dependent.
b ) Some entries end abruptly mid sentence (channel 7 even with some quite short descriptions) or end with an unecesary tag like [WS] or [Program info # Ten]. I think these are just not transmitted in their entirety but is it possible EPG table format has some sort of a more to follo extended entry?
c) Channel nine often have an obscure title like "Saturday night at the movies" as the title, and not the name of the actual movie in the title.
d) Entries for TV series now often have the series name AND the name of the episode in the title, making it annoying when adding to a regualr autoschedule like webscheduler, because the complete title (the defaultt action) only ever occurs once during the airing of the series. Is there something like a sub-title entry for the xml output format? (LOL not subtitles! but a secondary title entry!
e) some left over html tags like <br> (channel 9)

Most of this could be tidied up with a good text parsing tool (for windows!). Any suggestions? If I find a good one I would be prepared to have a stab at it.
Jeltz

This post has been edited by Jeltz: Mar 29 2008, 08:39 AM

[null_pointer]

Most of this could be tidied up with a good text parsing tool

There is already a function to do this using XSLT, check out the xslt directory, this has more info on how to use this with an example for channel 9.

[Jeltz]

Thanks, I remember the xslt stuff now you mention it.

I am very annoyed with channel 9 at the moment, who tonight (saturday) filled up their 9HD EIT with the 9SD program guide, i.e. NOT what was being broadcast on 9HD. So I ended up looking at the wrong thing.

Jeltz

[nate]

Thought I'd post this to give anyone using EPGScan a warning.

I've just found that the ABC seems to be broadcasting incorrect times in their EPG.

Running EPGScan in verbose 3 mode gives the following event for the 7:30 report

CODE

Parsing Event
Event ID: 0xa5b3
Start Time = 2008/03/31  09:30:00
Duration: 1860
Descriptors Loop Length: 0x102
Parsing Descriptor
Descriptor Tag : 4d
Descriptor Data Length : 188
ISO Lang Code : eng
Name Length : 15
Text 1st BYTE : 0x54
Name : The 7.30 Report
Text Length : 168
Text 1st BYTE : 0x54
Text : The ABC's flagship current affairs program, presented by Kerry O'Brien. In-depth coverage and analysis of the major daily news events - both domestic and international.
Parsing Descriptor
Descriptor Tag : 54
Descriptor Data Length : 2
Contect Descriptor (0x80)(0x0) Level1:8 Level2:0
Parsing Descriptor
Descriptor Tag : 50
Descriptor Data Length : 16
Parsing Descriptor
Descriptor Tag : 50
Descriptor Data Length : 16
Parsing Descriptor
Descriptor Tag : 50
Descriptor Data Length : 20
Parsing Descriptor
Descriptor Tag : 55
Descriptor Data Length : 4

This shows that the start time of the 7.30 Report is 09:30:00 (9:30am + 11hrs = 8:30pm). If day light savings had finished then the time would be correct, but it hasn't, so it's not.

null,
I took a look at the code of EPGScan and there's nothing wrong. Modified Julian Date dates don't know anything about timezone offsets, so it's just the ABC that's stuffed up. However, I had a thought of a possible way to get EGPScan to correct it but I can't try it at the moment because i don't have a capture card in my dev machine at the moment.
I don't know if it's being broadcast, but if the TOT table is being broadcast then the local_time_offset can be compared with the local computer's offset and any discrepency could be corrected.

[null_pointer]

I don't know if it's being broadcast, but if the TOT table is being broadcast then the local_time_offset can be compared with the local computer's offset and any discrepency could be corrected.

Its a good an idea but I am not sure it would help, in situations like this the local PC time/timezone/DST offset is just as likely to be completely screwed as well.

[tre30]

Microsoft have fixed the DST with an update, its been available for awhile, cumulative DST update - WindowsXP-KB942763-x86-ENU.exe (sorry no don't have direct link but search for that should find you it)

Java however does not get corrected and must be updated.
Now on too the issue at hand;
How do you use the .xsl template? I've given it a go on ch9 eit data from epgscan using both the java method and the msxml method (both producing the same results) - a somewhat corrected (still not sure about that) but totally xml <tag> devoid end product? Not sure if I'm doing something wrong or if ch9 has changed their style since you originally formed the sample-09.xsl template.

[null_pointer]

without seeing the input xml file the xsl transform file and the output file I can not really tell you what is happening. Perhaps zip and post the files or post extracts.

[SimonP]

Has anyone had this working in the UK? I want to extract the EPG data from our DVB-T transmissions for use with Web Scheduler but I can't get it to do anything. Maybe it doesn't work with "sub pic" text, only "teletext"?

If not, can anyone suggest something that'll work in the UK and supply WS with EPG data? I'm using XMLTV at the moment but would rather pull it directly from the transmitted data.

[null_pointer]

Linux Tool

http://www.darkskiez.co.uk/index.php?page=tv_grab_dvb

DVBGuide (UK Project)

http://www.geocities.com/pclareuk/DVBGuide/

[SimonP]

Thanks, DVBguide is working nicely, as is WS

[vex-box]

Ch9 seems to transmit all their EIT data in capital letters here in Darwin. Oh how surprising Ch9 going against the grain...
Would I be able utilised the XSLT of EPGScan to change it all to lower case with some capitals per normal punctuation?

Thanks Vex...

This post has been edited by vex-box: Jun 26 2008, 07:03 PM

[null_pointer]

Yes the XSLT can be used to do this.