zeroing out part of an SDcard

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

zeroing out part of an SDcard

Robert Moskowitz
More on my compressing images from SDcards.

I only need the boot info and 1st partition, so I remove the other
partitions, but of course there is still 1s out there so it will not
compress efficiently. I want to zero out the end of the card so first I
use fdisk:

# fdisk -l /dev/sdb

Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0009e2ad

Device Boot Start End Blocks Id System
/dev/sdb1 8192 1007615 499712 83 Linux

Then dd:

# dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620
dd: ‘/dev/sdb’: cannot seek: Invalid argument
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.000652852 s, 0.0 kB/s

======================

seems you cannot seek on an SDcard. How do I do this? Since the change
in partition 1 is trivial, if I have to start again from an SDcard with
all 3 paritions, I can do that...


--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Robert P. J. Day-2
On Thu, 21 Aug 2014, Robert Moskowitz wrote:

> More on my compressing images from SDcards.
>
> I only need the boot info and 1st partition, so I remove the other partitions,
> but of course there is still 1s out there so it will not compress efficiently.
> I want to zero out the end of the card so first I use fdisk:
>
> # fdisk -l /dev/sdb
>
> Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Disk identifier: 0x0009e2ad
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 8192 1007615 499712 83 Linux
>
> Then dd:
>
> # dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620
> dd: ‘/dev/sdb’: cannot seek: Invalid argument
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.000652852 s, 0.0 kB/s
>
> ======================
>
> seems you cannot seek on an SDcard. How do I do this? Since the change in
> partition 1 is trivial, if I have to start again from an SDcard with all 3
> paritions, I can do that...
  it seems like you're trying to seek more than a terabyte into that
card.

rday

--

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================
--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Stephen Berg (Contractor)-2
In reply to this post by Robert Moskowitz
On 08/21/2014 07:51 AM, Robert Moskowitz wrote:

> More on my compressing images from SDcards.
>
> I only need the boot info and 1st partition, so I remove the other
> partitions, but of course there is still 1s out there so it will not
> compress efficiently. I want to zero out the end of the card so first
> I use fdisk:
>
> # fdisk -l /dev/sdb
>
> Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Disk identifier: 0x0009e2ad
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 8192 1007615 499712 83 Linux
>
> Then dd:
>
> # dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620
> dd: ‘/dev/sdb’: cannot seek: Invalid argument
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.000652852 s, 0.0 kB/s
>
> ======================
>
> seems you cannot seek on an SDcard. How do I do this? Since the change
> in partition 1 is trivial, if I have to start again from an SDcard
> with all 3 paritions, I can do that...
>
>
I'd create a new partition on the remainder of the card, and then dd
/dev/zero to that partition.  I don't know squat about the bits and
pieces of partition structure but that should wipe most of it at least.

--
Stephen Berg
Systems Administrator
NRL Code: 7320
Office: 228-688-5738
[hidden email]

--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Robert Moskowitz
In reply to this post by Robert P. J. Day-2

On 08/21/2014 09:07 AM, Robert P. J. Day wrote:

> On Thu, 21 Aug 2014, Robert Moskowitz wrote:
>
>> More on my compressing images from SDcards.
>>
>> I only need the boot info and 1st partition, so I remove the other partitions,
>> but of course there is still 1s out there so it will not compress efficiently.
>> I want to zero out the end of the card so first I use fdisk:
>>
>> # fdisk -l /dev/sdb
>>
>> Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors
>> Units: sectors of 1 * 512 = 512 bytes
>> Sector size (logical/physical): 512 bytes / 512 bytes
>> I/O size (minimum/optimal): 512 bytes / 512 bytes
>> Disklabel type: dos
>> Disk identifier: 0x0009e2ad
>>
>> Device Boot Start End Blocks Id System
>> /dev/sdb1 8192 1007615 499712 83 Linux
>>
>> Then dd:
>>
>> # dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620
>> dd: ‘/dev/sdb’: cannot seek: Invalid argument
>> 0+0 records in
>> 0+0 records out
>> 0 bytes (0 B) copied, 0.000652852 s, 0.0 kB/s
>>
>> ======================
>>
>> seems you cannot seek on an SDcard. How do I do this? Since the change in
>> partition 1 is trivial, if I have to start again from an SDcard with all 3
>> paritions, I can do that...
>    it seems like you're trying to seek more than a terabyte into that
> card.

Well, man says about seek:

        seek=N skip N obs-sized blocks at start of output

and obs defaults to 512 bytes, and I *THOUGHT* fdisk was reporting # 512
blocks, not bytes.

--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Robert P. J. Day-2
On Thu, 21 Aug 2014, Robert Moskowitz wrote:

>
> On 08/21/2014 09:07 AM, Robert P. J. Day wrote:
> > On Thu, 21 Aug 2014, Robert Moskowitz wrote:
> >
> > > More on my compressing images from SDcards.
> > >
> > > I only need the boot info and 1st partition, so I remove the other
> > > partitions,
> > > but of course there is still 1s out there so it will not compress
> > > efficiently.
> > > I want to zero out the end of the card so first I use fdisk:
> > >
> > > # fdisk -l /dev/sdb
> > >
> > > Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors
> > > Units: sectors of 1 * 512 = 512 bytes
> > > Sector size (logical/physical): 512 bytes / 512 bytes
> > > I/O size (minimum/optimal): 512 bytes / 512 bytes
> > > Disklabel type: dos
> > > Disk identifier: 0x0009e2ad
> > >
> > > Device Boot Start End Blocks Id System
> > > /dev/sdb1 8192 1007615 499712 83 Linux
> > >
> > > Then dd:
> > >
> > > # dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620
> > > dd: ‘/dev/sdb’: cannot seek: Invalid argument
> > > 0+0 records in
> > > 0+0 records out
> > > 0 bytes (0 B) copied, 0.000652852 s, 0.0 kB/s
> > >
> > > ======================
> > >
> > > seems you cannot seek on an SDcard. How do I do this? Since the change in
> > > partition 1 is trivial, if I have to start again from an SDcard with all 3
> > > paritions, I can do that...
> >    it seems like you're trying to seek more than a terabyte into that
> > card.
>
> Well, man says about seek:
>
>        seek=N skip N obs-sized blocks at start of output
>
> and obs defaults to 512 bytes, and I *THOUGHT* fdisk was reporting # 512
> blocks, not bytes.
  if memory serves, using "bs=" overrides both ibs= and obs= ...
here's from the man page:

       bs=BYTES
              read and write up to BYTES bytes at a time

notice that says "read *and* write", so i'm simply assuming it is the
value used for both ibs and obs.

rday

--

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================
--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Chris Adams-2
In reply to this post by Robert Moskowitz
Once upon a time, Robert Moskowitz <[hidden email]> said:
> >># dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620
<snip>
> Well, man says about seek:
>
>        seek=N skip N obs-sized blocks at start of output
>
> and obs defaults to 512 bytes, and I *THOUGHT* fdisk was reporting #
> 512 blocks, not bytes.

fdisk reports sectors (that's the default in recent years, it can be
configured to report in different units).  However, look again at the dd
command; the block size was set to 1M (=1048576 bytes), and then the
seek was set to 1007620 blocks, which is almost 1TB.  If you are going
to raise the block size, you have to recalculate everything else that is
expressed in blocks.
--
Chris Adams <[hidden email]>
--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Robert Moskowitz

On 08/21/2014 09:50 AM, Chris Adams wrote:

> Once upon a time, Robert Moskowitz <[hidden email]> said:
>>>> # dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620
> <snip>
>> Well, man says about seek:
>>
>>         seek=N skip N obs-sized blocks at start of output
>>
>> and obs defaults to 512 bytes, and I *THOUGHT* fdisk was reporting #
>> 512 blocks, not bytes.
> fdisk reports sectors (that's the default in recent years, it can be
> configured to report in different units).  However, look again at the dd
> command; the block size was set to 1M (=1048576 bytes), and then the
> seek was set to 1007620 blocks, which is almost 1TB.  If you are going
> to raise the block size, you have to recalculate everything else that is
> expressed in blocks.
Ah...

I had cobbled this together from instructions on how to zero out a whole
drive.  That is were the bs= comes from.  So this overrides the 512bytes
with a much bigger number, making the seek= wrong.  Got it.  Will try again.


--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Chris Murphy
In reply to this post by Robert Moskowitz

On Aug 21, 2014, at 6:51 AM, Robert Moskowitz <[hidden email]> wrote:

> More on my compressing images from SDcards.
>
> I only need the boot info and 1st partition, so I remove the other partitions, but of course there is still 1s out there so it will not compress efficiently. I want to zero out the end of the card so first I use fdisk:
>
> # fdisk -l /dev/sdb
>
> Disk /dev/sdb: 7.4 GiB, 7969177600 bytes, 15564800 sectors
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Disk identifier: 0x0009e2ad
>
> Device Boot Start End Blocks Id System
> /dev/sdb1 8192 1007615 499712 83 Linux

These 487MB aren't zero'd, so whatever is reported as free space by the file system will have stale data on its sectors. Mount sdb1 and use fstrim -v <mountpoint> and see if the command is accepted.

>
> Then dd:
>
> # dd if=/dev/zero of=/dev/sdb bs=1M seek=1007620

It's kinder to your sdcard to create a 2nd partition for the remaining space and trim it. This can be done with either mkfs.btrfs or mkfs.ext4, if the kernel detects it's solid state then both commands issue trim for all of those blocks, and then creates a file system. I'd use btrfs only because it writes out far less metadata.

However, quite a few sdcards don't respond to trim. And if they do, some have non-deterministic read of 'erased/trimmed' pages, in which case even though they're marked as erased, they can still return old data or garbage. And in that case unfortunately zeroing is the only way out.


Chris Murphy
--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Chris Adams-2
Once upon a time, Chris Murphy <[hidden email]> said:
> However, quite a few sdcards don't respond to trim.

Do _any_ SD cards support TRIM?  I didn't think that was supported by
the protocol at all.
--
Chris Adams <[hidden email]>
--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Chris Murphy

On Aug 21, 2014, at 11:00 AM, Chris Adams <[hidden email]> wrote:

> Once upon a time, Chris Murphy <[hidden email]> said:
>> However, quite a few sdcards don't respond to trim.
>
> Do _any_ SD cards support TRIM?  I didn't think that was supported by
> the protocol at all.

It's a good question. fstrim spit back a message to the effect the device doesn't support trim. When I mkfs.btrfs, however, it reports trimming. And then when I dd read LBA's known to have previously contained data, I get zeros. *shrug* I don't actually know what's going on.

And then there's this:
https://www.sdcard.org/downloads/formatter_4/

"Using generic formatting utilities may result in less than optimal performance for your memory cards." So that implies they have something like trim, even if not the ATA variety. So I'm curious what this formatter is doing, and I'm also curious if we have an equivalent on linux or if in fact we get it already with normal mkfs; but then if that's true why does fstrim fail to do it.


Chris Murphy
--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Chris Murphy
SD Specifications
Part 1
Physical Layer
Simplified Specification
Version 4.10
https://www.sdcard.org/downloads/pls/simplified_specs/part1_410.pdf

page 38

4.3.5 Erase
It is desirable to erase many write blocks simultaneously in order to enhance the data throughput. Identification of these write blocks is accomplished with the ERASE_WR_BLK_START (CMD32), ERASE_WR_BLK_END (CMD33) commands.
The host should adhere to the following command sequence: ERASE_WR_BLK_START, ERASE_WR_BLK_END and ERASE (CMD38).


So there's something like trim, but trim is an ATA command. But I don't know whether this is what mkfs.btrfs is using, or if it's asking some other library to do it, and it's that library that differentiates and mkfs.btrfs has no idea. And then whether fstrim is using some other library and hence the difference.

Rabbit hole.


Chris Murphy
--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Chris Murphy
PNY SDHC card, 16GB - pretty generic. In this case fstrim did work (pfft yeah whatever it didn't the last time I tried it), and so did mkfs.btrfs but they work differently according to strace.

fstrim issues:
ioctl(3, FITRIM, 0x7fffbf6b87e0)        = 0

The result is
write(1, "/mnt/: 13.9 MiB (14598144 bytes)"…, 41/mnt/: 13.9 MiB (14598144 bytes) trimmed

And this is roughly the amount of data deleted during the current mount. It's a 13GB partition, so fstrim is definitely not going to assure you of erasing previously unerased blocks. Just what the fs is aware of having been recently deleted. LBA's with known data in them still had data in them after fstrim, even though they were long ago deleted files from a completely different fs volume.

mkfs.btrfs issues:
write(2, "Performing full device TRIM (13."..., 43Performing full device TRIM (13.72GiB) ...
) = 43
ioctl(3, BLKDISCARD, {0, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {40000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {80000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {c0000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {100000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {140000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {180000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {1c0000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {200000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {240000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {280000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {2c0000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {300000000, 7fff920ee4a0}) = 0
ioctl(3, BLKDISCARD, {340000000, 7fff920ee4a0}) = 0

And this obliterated everything on that partition. A dd read came up with 99.99% zeros, the rest of it was just the btrfs superblocks.

I wasn't successful using scsi kernel tracing to find out what commands are actually sent to the sdcard for either of these. The trace_pipe file only showed the ssd events, not for /dev/mmcblk0 and I'm not sure how to trace it.

But based on available info I'm pretty sure it's not actually getting issued TRIM in the ATA sense, but it's probably getting the SD Card equivalent which seems to be the ERASE_* set of commands.


Chris Murphy
--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org
Reply | Threaded
Open this post in threaded view
|

Re: zeroing out part of an SDcard

Chris Murphy

Nice.

http://www.spinics.net/lists/linux-btrfs/msg36918.html

No special tool needed. e.g. I just tested this with Gnome Disks, with the default "don't overwrite with zeros (quick format)" option, using btrfs. All of, and only that chosen partition, was completely erased in about 2 seconds. Every sector returns FF's [1] other than btrfs superblocks.

I did a strace on mkfs.xfs, mkfs.ext4, and mkfs.vfat -F32; but did not test this in Gnome Disks. The strace reveals that both xfs and ext4 use BLKDISCARD, while vfat does not. xfs and ext4 further makes use of:

ioctl(3, BLKDISCARDZEROES, 0)           = 0

I'm not sure what this does. Maybe it causes erased blocks to deterministically return zeros instead of FF's?

Anyway, it looks like SD cards are in effect "trimmed" by formatting them ext4, xfs or btrfs, as long as the tool being used is using the fs's mkfs command, and also doesn't pass a nodiscard option.

I didn't test mkfs.ntfs or mkfs.hfs, but like vfat if any of them aren't calling BLKDISCARD it's probably a valid feature request for their upstream. Meanwhile a workaround is to use mkfs.btrfs with no options, and then wipefs -a that same partition, and then format with the preferred filesystem.


Chris Murphy



[1]
[root@twenty1 chris]# dd if=/dev/mmcblk0p2 bs=1M skip=313 | hexdump -C
00000000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
--
users mailing list
[hidden email]
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org