MTR 6 ANNOUNCEMENTS - READ FIRST!

[LilAbner]

ok , i stripped and ceaned all drives and player to rule out dirty optics.
No difference , nothing can play it ,so im going to mark it as broken.

I need to run the error correction skip code a few times on disks that are known good with errorData on the disk , just to cinfirm the c=new method is working correctly.
Tie up a few loose ends because of the new code ( mostly UI related )

thenI can,hopefully package this up...

GB, What is the name of that disk?

[geezerbuttz]

Flight Of The Living Dead

[LilAbner]

Flight Of The Dead

I can not find any reference for that one

[geezerbuttz]

check name i missed a word .. duh!

[geezerbuttz]

Im running this deep scanning code. I made more changes to assist in correct cell skipping , if we cannot calculate a skip , we will drop to 1 sector nibble.

It looking like this can be used to read as much data that you can from a defective disk.
As long as the main ifo is available , we can calculate what we need.

So far the disk that raised this issue , is getting run again , and more data than ever is being recovered.
It seems its the ability for the disk to reflect light that has failed....


So much for 'lasting a lifetime" eh....

[TheSystemHasFailed]

It seems its the ability for the disk to reflect light that has failed....


So much for 'lasting a lifetime" eh....

Yep. low quality reflective material used in the coating and/or low quality sealing that lead to oxidation creeping in..

Absolutely terrible when that happens..

[geezerbuttz]

All looking good , tomorrow I will add a "bit level copy" button in the ISO panel , that will set the copy to a raw 1 to 1 copy , with no blocks missed , or calculated jumps...

Might as well , the ability is there.

[geezerbuttz]

State Of Play:

as you know i have been working on a low level reader for extremely corrupted disks.
Its going well , untill we hit areas that I cannot successfully get any correct data to help me navigate further.

In one case I have no IFO at the location indicated by the SRPT , and nothing in the bytes that can be read to assist in speeding up the process.

What i currently have is , if 1 sector cannot be read , advance 15 forward , try again , if not , blank 16 packs, else drop down into 1 sector mode , and nibble.

This still takes time in very bady mastered areas, or corrupt areas.

Fermenting in my brain is this:
On Linux , i ve userland access to the DVDDrive , and I can disable is read reties.
If we do that , and read each sector , we can build up a raw map , much quicker ( assumption currently) We can then load that into MTR and have it convert that into MIF map ready to save back out or use.

Not sure about this yet , as its only in the idea stage , but if I can pull that off , you could do this in a VM on the mac , or a dedicated drive in a PC/Mac configured as Linux OS
G

[geezerbuttz]

ok ..

So my drive has its error retries set to 128... yes , 128 tries per bad sector...
So , in linux , it was trivial to set that to 1 , then run a dd on the disk ignoring readerrors.
It goes through quite nicely.

Apple treats us like kids , not allowing us access to the drives in userland.
You can do it , but you need to provide a kernel as well that supports the commands you need to execute , and no way is apple going to sign off on a kernel that allows you to reduce the read/write security of a device.

So , Linux is the way forward, you can do this in a virtual environment on the mac.
I just installed linux mint....
Im looking to get objC into linux ( it can be done ) then I will write a small command line app to get the raw structure data for mtr ... MTR can then do a conversion to a MIF.

Need to iron out a few things first..

[geezerbuttz]

Just playing while this maps out the disk... and installed GNUStep in linux.

It's basically nextStep. we have project builder , that if anybody remembers is the same as the first Devkit for OSX.
Boy we are spoilt these days....

But , i got a test piece of code compiled and running under Linux.
You will need to install GNUStep to run it though , but thats simple enough form the command line.


This is interesting , As we will be able to generate Deep ISO images faster than ever before.
Right now , it still indicates a day to image , but it based this on every sector being corrupt , and that is not the case.

I will play around tuning the SCSI controls to see if we can get it any faster by disabling EEC correction as well, but im not keen on that Idea.

G

[geezerbuttz]

right , its mapping in linux..
Instead of 3 mins , its seven seconds to recover from a read error.

Tomorrow , I will start to work out the class I need to store the sectors , mimic a little code from MTR to do a basic alignment , then see what options in the older objC I have to write out...

Then we have to write a loader/convertor in MTR.

Its all go...
( even with this improvement , heavily damaged / obfuscated disks will be an over nighter.)

[geezerbuttz]

this morning i have been working on how to accelerate the reading of corrupt blocks.

Reading 1 block , even with ErrorCorrection disabled takes 7 seconds at worst.
So 16 sectors takes just under 2 minutes...

This is not good enough of devices with masses of corruption.
So i implemented a read algo that checks a group of blocks that are no bigger than the minimum file size a Disk may hold..

so far , i have got it down to 30 seconds for 32 sectors ( it was 4 minutes... )

[geezerbuttz]

spent the afternoon doing some long analysis of the code that samples defective areas , and decided that it wastes too much time repeating blocks that have already been scanned.

It also had a potential ( proven ) that it would spend a lot of time still reading single blocks.

Adapted a divide and conquer approach...

Should minimise the total of bad sectors encountered , and reduce the amount of times we re enter the bad block scanning functions.

Testing now.

[geezerbuttz]

Well , that was a total PITA.

The conversion for The linux app , to a structured replication of the disk format , when imported back into MTR was a nightmare.

Managed to get there though.
I need to generate a few maps from the Linux app to fully test the data exchange , but if that passes ...we are good to go.
G

[geezerbuttz]

Everything works , but to assist the linux app , and make it a little more faster , Im in the process of porting across my IFO loader framework , as we can use the to check our location on the disk to assist in a faster jump routine....

I only have to remove the decryption calls , and replace the load, read and seek with standard C implementations.

hey , it takes time , but its better I give you all the tools to do the job.
It will bring the linux app to the same state as the deep mapper in ISO mode... but faster due to the ability to simply walk a defective sector in 6 seconds , verses 3 minutes.

Keep you hair on...

[TheSystemHasFailed]

Keep you hair on...

too late! Lost that years ago...

It's fascinating to watch your development process tho as you explain it.. sounds like MTR 6 will be a monster app when it's done...

[geezerbuttz]

Its cool , but first time i provided a linux front end to map the disk.

Thats new

[geezerbuttz]

ok , i have been quiet for bit i know:

I have been working on the linux core code , It works , but it has revealed some issues that needed to be addressed.

I was successfully able to nibble single sectors of the disk , getting a good map layout, but when grouping those readable sectors back together in MTR , it would fail to read a group of blocks that previously had passed.

With that I wrote a verification routine in the linux app , and that yet again , when reading a group of sectors would also fail .

After a lot of observation , it seems that if the drive tries to read data faster than it is designed to do , or the OS requests a faster speed than the medium can handle , the drive will start to report errors on the larger block read.

In the linux app I have enabled the ability to set the read speed , and have found that a flat x4 is the sweet spot for this.

Of course , in MTR I have to implement the same now , and Im not sure how to go about that ( ioctrls i guess )

On A side not I installed ( after much swearing ) linux Mint on a seperate HD yesterday to boot for , instead of running virualy. First thing I found after installing GNUstep ( open step replacement ) was that the frameworks are different , and there fore I needed to re compile the app. Next , I found that the recovery of the DVDDrive had dropped to 12 seconds , verses the VM linux mint install of 6seconds.

So , instal linux in a VM for this , it seems to have better recovery performance. Of course this could just be the Kernel in the newer version of mint.

So , im testing a nasty right now....

What we now have is a triple layer of jump assistance in the linux app.
First we look at the Ifo , and see if we can calculate the next EndOfCell Jump. If that fails we check the SRI table store , and see if we can jump over towards the end of cell ( SRI table is relative withing the current cell ) , if that fails we drop down to reading 16 blocks , then , once we can read again , rewind back until we fail again.

This gives us a good approximation.


I can later give an option , to reprocess Corrupt blocks in a single sector probe to give any sectors within them change to be read.
That of course will take a very long time.....

.. The last point , is , Im thinking to port this accross to the Mac , minus the ability to disable drive retries , as we cant do that.

The overall code is very good....

All these apps will be included in the MTR Family Package , that will come with the installer / download.

G

[geezerbuttz]

well , here we go , last entry for a week , as I leave for home tomorrow to bury my father.

read():

This is a strange one , when requesting (n) bytes to be read , read() should return (n)bytes or (x)bytes that were available.
In Linux it behaves exactly as expected.

In MacOS , using the same code , when it cannot read when was requested it simply fails. (-1)

That is not what we should get.

Im going to look at the other binary reading methods we have to see if we can get the expected result.
G

[geezerbuttz]

So: before I go to bed, MacOS does not return ( from what I have seen ) partial reads from the sys_calls.

I tried:

read().
fread().
pread().
NSFileHandler

none returned partial reads.
This is most likely the sys_call in the kernel , that underwrites most of these , does not do it either.
Bugger.

I have had to write my own , and that slows don the app somewhat , but it replicates the Linux read() sys_call.

That means we read blocks otherwise the mac would miss.

[TheSystemHasFailed]

I have had to write my own , and that slows don the app somewhat , but it replicates the Linux read() sys_call.

That means we read blocks otherwise the mac would miss.

So just curious.. were you able to 'borrow' code from the linux code source to replicate this (I mean so much of it is open source after all and GPL v2). Or did you have to resort to only using it for 'inspiration'...?

[geezerbuttz]

Rolled my own , but performance is pretty much the same.

When I get back home , I intend to update that with a hybrid switch.

When it detects a read error , it drops into partial mode , and stays there until it it reads (n) full reads , when it switches back into full read.


That way we get the speed gains of the larger block reading , and only 1 extra corrupt retry cycle on initial change to partial read mode.

More soon.

[TheSystemHasFailed]

Rolled my own , but performance is pretty much the same.

I've been coding a long time.. my first computer was a ZX81 (remember those?)

And I've used *nix for a long time.. I would be hard pressed to roll my own version of a read sys_call without doing a fair bit of research and reading of code..

When I grow up I want to be like you....

[rthpjm]

Rolled my own , but performance is pretty much the same.

I've been coding a long time.. my first computer was a ZX81 (remember those?)

Me too!
Ah, the good old days… waiting for the tech press magazine (printed paper magazine, of course) to arrive. Trying out the Z80 machine code example game. We didn’t need compilers back then.

And I've used *nix for a long time.. I would be hard pressed to roll my own version of a read sys_call without doing a fair bit of research and reading of code..

Me too!!

When I grow up I want to be like you....

Me too!!!

[geezerbuttz]

ZX80
ZX81
TI-99-4A
Atari 130XE
Atari 520ST
Amiga 500
Amiga 2000
Mac 650 Centris
Mac 7300 ( with accelerator! whoooo )
Original blue white power mac ,
PPC MDD
Several power macs that failed costing me a fortune ( thanks apple for crap water cooling )
Mac mini , iMacs , power books of many assortments....

Hackintoshes.

( Assembly on the ATMEL 328 line of embedded , but thats another story )

I have been around.

[geezerbuttz]

Back to the Job:

The macs read function got a serious update:

It switched between bulk read and nibble read .
When (n) number of sectors are read concistantly , we switch to reading large chnks in one go into the buffer , instaed of filling the buffer with single sector reads.

On encountering a group of sectors that cannot be read , it drops back into reading single sectors , filling the buffer with what it can read , then processing the calculation for the next possible readable sector.

Once it starts to read consistently again , it switches gears , and the speed can increase dramatically.

The linux app , got some more updates :
The code to prude blocks that have partial reads was improved from MacOS version.
All UI elements of the linux app have been designed to match the MacOS one as closely as possible , thus no more "windows 95 look and feel"

The verify stage has a custom progress that reveals block states:
Green = Original READ_OK and verified READ_OK.
YELLOW = Originally READ_ERROR and verified READ_ERROR
ORANGE = Original READ_OK but verification failed to seek to the start address.
RED = Original READ_OK but verification failed to read all or part of the required block.

The last two entries will update the structure to make the block as invalid before you save out.
These cases are most likely an indication of failing media or failing drive.

I am now testing the Mac side, the Linux has passed ok.

I need to remove the last code I added in MTR as this DVDSectorScan app now does this job better.
You save a .dvdss file from these apps that can be imported into MTR , there you can use as is , export to .MIF so they can be included in the file reserve.

It a complete suite of apps.

G

[geezerbuttz]

Hi All.

So , i upgraded the progress and verification displays again.

Firstly its hard to display 4000000 sectors in a bar that has a max resolution of 450 pixels across.
This lead to very small areas of defective data being to small to be represented in the display.

I countered this by calculating 1% in sector space and 1% in graphic space. Normaly the bar is blue , but the 1% in bar space changes to match its error color.
This way the bar can show all conditions.

Next , and very stupid of me , i was calculating all the progress draw calculations in the main draw function.
This way every time the bar was drawn , a mass of calculations were performed.

I moved the calculation into the initial creation , and now store everything , including the BezierPath in a class object.
That way we only have to fast enumerate the array that holds the class objects , pulling the colour and path to be displayed... no more calculations.


This is now done in both apps for Linux and MacOS.

The sector scanners are complete.

Tomorrow I start testing, I still need to disable the prototype I did in MTR of the sector scanner and remove it, but as far as I am concerned stage one is complete.

So give me a few days to test , clean , final build and package , and i hope to have it hit the update stream next week.
G

[geezerbuttz]

small bug in the output phase.
Corrected that.

Added auto device change notification to both apps , got fed up pressing refresh.

Back to testing.

G

[geezerbuttz]

Updated the main menu dynamic code to remove a lot of #ifdefs..

Power User Tip:
If you extract an ISO and need to test it , dont play it, load it back into MTR and ( if licensed ) run it in DVDSectorProbe.
If probe exits prematurely , then its not good. If it runs to complete , its good.

Back to testing.

[geezerbuttz]

ok , shared workspace notifications do not work in Linux.
So no auto disk mount notifications.

This may be down to Linux being a virtual machine , or just Gnustep has not fully implemented device mount/unmount
notifications...

We will see in a later build.

Regarding that, Gnustep versions vary on different distros , so you will need to request that I build a version for your Distro.

Its not a problem as long I can install it virtually.
We will store the different build locally here.