Transferring files over tftp

Transferring files over tftp

[Teresa Remmet]

Hello,

I am facing tftp issues when trying to transfer files > 2GB. This ends up in a crash:

barebox@Phytec phyCORE AM335x:/ cp /mnt/tftp/2GB_plus_1M .
eth0: 100Mbps full duplex link detected
T T T T T T T T T T T T T T read: Connection timed out
unable to handle paging request at address 0x200d319a
pc : [<9fe0587a>]    lr : [<9fe058cb>]
sp : 9ffef810  ip : ffffffff  fp : 9004d4d0
r10: 00000000  r9 : 9004e410  r8 : 00000006
r7 : 00000000  r6 : 8fefeec0  r5 : a79cdbf6  r4 : 200d318e
r3 : 0000001a  r2 : 00000019  r1 : 9004f40c  r0 : 8fefeec0
Flags: nzCv  IRQs off  FIQs on  Mode SVC_32
[<9fe0587a>] (remove_free_block+0xe/0x3e) from [<9fe058cb>] (block_remove+0x21/0x24)
[<9fe058cb>] (block_remove+0x21/0x24) from [<9fe058eb>] (block_merge_next+0x1d/0x34)
[<9fe058eb>] (block_merge_next+0x1d/0x34) from [<9fe05b73>] (tlsf_free+0x55/0x62)
[<9fe05b73>] (tlsf_free+0x55/0x62) from [<9fe408e7>] (copy_file+0x77/0x13c)
[<9fe408e7>] (copy_file+0x77/0x13c) from [<9fe33c41>] (do_cp+0x101/0x124)
[<9fe33c41>] (do_cp+0x101/0x124) from [<9fe0318d>] (execute_command+0x21/0x48)
[<9fe0318d>] (execute_command+0x21/0x48) from [<9fe07c81>] (run_list_real+0x5a9/0x608)
[<9fe07c81>] (run_list_real+0x5a9/0x608) from [<9fe075dd>] (parse_stream_outer+0x105/0x1
5c)
[<9fe075dd>] (parse_stream_outer+0x105/0x15c) from [<9fe07ea3>] (run_shell+0x33/0x60)
[<9fe07ea3>] (run_shell+0x33/0x60) from [<9fe0318d>] (execute_command+0x21/0x48)
[<9fe0318d>] (execute_command+0x21/0x48) from [<9fe07c81>] (run_list_real+0x5a9/0x608)
[<9fe07c81>] (run_list_real+0x5a9/0x608) from [<9fe0796d>] (run_list_real+0x295/0x608)

[<9fe5603d>] (unwind_backtrace+0x1/0x58) from [<9fe00cf1>] (panic+0x1d/0x34)
[<9fe00cf1>] (panic+0x1d/0x34) from [<9fe543ad>] (do_exception+0xd/0x10)
[<9fe543ad>] (do_exception+0xd/0x10) from [<9fe5440d>] (do_data_abort+0x21/0x2c)
[<9fe5440d>] (do_data_abort+0x21/0x2c) from [<9fe54134>] (do_abort_6+0x48/0x54)

Copying a file <2GB works even without timeouts.

Causing this seems to be commit:

commit e3e55fa3aa698bb92fa4362a36e102749c13c124
Author: Sascha Hauer <s.hauer@pengutronix.de>
Date:   Thu Aug 30 12:27:00 2018 +0200

    fs: tftp: improve file size handling
    
    Previously we used FILE_SIZE_STREAM unconditionally. Instead, fill the
    inode size with a valid filesize if we have one and only if not fall
    back to FILE_SIZE_STREAM.

Reverting the commit reveals a second issue. After 2GB the transfer breaks up silently.
Maybe some overflow? Any idea where to look at this further?

Thanks,
Teresa

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: Transferring files over tftp

[Sascha Hauer]

Hi Teresa,

On Thu, Nov 15, 2018 at 04:24:18PM +0100, Teresa Remmet wrote:
> Hello,
> 
> I am facing tftp issues when trying to transfer files > 2GB. This ends up in a crash:
> 
> barebox@Phytec phyCORE AM335x:/ cp /mnt/tftp/2GB_plus_1M .
> eth0: 100Mbps full duplex link detected
> T T T T T T T T T T T T T T read: Connection timed out
> unable to handle paging request at address 0x200d319a
> pc : [<9fe0587a>]    lr : [<9fe058cb>]
> sp : 9ffef810  ip : ffffffff  fp : 9004d4d0
> r10: 00000000  r9 : 9004e410  r8 : 00000006
> r7 : 00000000  r6 : 8fefeec0  r5 : a79cdbf6  r4 : 200d318e
> r3 : 0000001a  r2 : 00000019  r1 : 9004f40c  r0 : 8fefeec0
> Flags: nzCv  IRQs off  FIQs on  Mode SVC_32
> [<9fe0587a>] (remove_free_block+0xe/0x3e) from [<9fe058cb>] (block_remove+0x21/0x24)
> [<9fe058cb>] (block_remove+0x21/0x24) from [<9fe058eb>] (block_merge_next+0x1d/0x34)
> [<9fe058eb>] (block_merge_next+0x1d/0x34) from [<9fe05b73>] (tlsf_free+0x55/0x62)
> [<9fe05b73>] (tlsf_free+0x55/0x62) from [<9fe408e7>] (copy_file+0x77/0x13c)
> [<9fe408e7>] (copy_file+0x77/0x13c) from [<9fe33c41>] (do_cp+0x101/0x124)
> [<9fe33c41>] (do_cp+0x101/0x124) from [<9fe0318d>] (execute_command+0x21/0x48)
> [<9fe0318d>] (execute_command+0x21/0x48) from [<9fe07c81>] (run_list_real+0x5a9/0x608)
> [<9fe07c81>] (run_list_real+0x5a9/0x608) from [<9fe075dd>] (parse_stream_outer+0x105/0x1
> 5c)
> [<9fe075dd>] (parse_stream_outer+0x105/0x15c) from [<9fe07ea3>] (run_shell+0x33/0x60)
> [<9fe07ea3>] (run_shell+0x33/0x60) from [<9fe0318d>] (execute_command+0x21/0x48)
> [<9fe0318d>] (execute_command+0x21/0x48) from [<9fe07c81>] (run_list_real+0x5a9/0x608)
> [<9fe07c81>] (run_list_real+0x5a9/0x608) from [<9fe0796d>] (run_list_real+0x295/0x608)
> 
> [<9fe5603d>] (unwind_backtrace+0x1/0x58) from [<9fe00cf1>] (panic+0x1d/0x34)
> [<9fe00cf1>] (panic+0x1d/0x34) from [<9fe543ad>] (do_exception+0xd/0x10)
> [<9fe543ad>] (do_exception+0xd/0x10) from [<9fe5440d>] (do_data_abort+0x21/0x2c)
> [<9fe5440d>] (do_data_abort+0x21/0x2c) from [<9fe54134>] (do_abort_6+0x48/0x54)
> 
> Copying a file <2GB works even without timeouts.
> 
> Causing this seems to be commit:
> 
> commit e3e55fa3aa698bb92fa4362a36e102749c13c124
> Author: Sascha Hauer <s.hauer@pengutronix.de>
> Date:   Thu Aug 30 12:27:00 2018 +0200
> 
>     fs: tftp: improve file size handling
>     
>     Previously we used FILE_SIZE_STREAM unconditionally. Instead, fill the
>     inode size with a valid filesize if we have one and only if not fall
>     back to FILE_SIZE_STREAM.
> 
> Reverting the commit reveals a second issue. After 2GB the transfer breaks up silently.
> Maybe some overflow? Any idea where to look at this further?

As a first step try the attached patch which makes the filesize in tftp
a 64bit type so that we at least can handle the filesize numbers.

I tried transferring a file > 2GiB here and it seemed to work, although
I don't have a board with that amount of memory, so I had to write to
/dev/null. I don't know if that makes a difference.

Also you probably begin to hit the limits of the tftp servers. I
realized that our tftp server (atftpd) only reports the filesize for
files smaller than 2GiB, with bigger files it only reports '0'.

I think other tftp servers begin to make problems when the 16bit block
counter overflows, so with working 2GiB files you are lucky already ;)

Sascha

--------------------------------------8<----------------------------------

From bf3de3391c958c9c44fcfa54b2c1a6a29e68a5a4 Mon Sep 17 00:00:00 2001
From: Sascha Hauer <s.hauer@pengutronix.de>
Date: Fri, 16 Nov 2018 08:24:06 +0100
Subject: [PATCH] fs: tftp: Make filesize a 64bit type

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
---
 fs/tftp.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/fs/tftp.c b/fs/tftp.c
index 0d9ee6effd..8131ab27b3 100644
--- a/fs/tftp.c
+++ b/fs/tftp.c
@@ -80,7 +80,7 @@ struct file_priv {
 	int state;
 	int err;
 	char *filename;
-	int filesize;
+	loff_t filesize;
 	uint64_t resend_timeout;
 	uint64_t progress_timeout;
 	struct kfifo *fifo;
@@ -136,7 +136,7 @@ static int tftp_send(struct file_priv *priv)
 				"timeout%c"
 				"%d%c"
 				"tsize%c"
-				"%d%c"
+				"%lld%c"
 				"blksize%c"
 				"1432",
 				priv->filename + 1, 0,
@@ -235,7 +235,7 @@ static void tftp_parse_oack(struct file_priv *priv, unsigned char *pkt, int len)
 		if (val > s + len)
 			return;
 		if (!strcmp(opt, "tsize"))
-			priv->filesize = simple_strtoul(val, NULL, 10);
+			priv->filesize = simple_strtoull(val, NULL, 10);
 		if (!strcmp(opt, "blksize"))
 			priv->blocksize = simple_strtoul(val, NULL, 10);
 		pr_debug("OACK opt: %s val: %s\n", opt, val);
@@ -658,7 +658,7 @@ static struct dentry *tftp_lookup(struct inode *dir, struct dentry *dentry,
 	struct fs_device_d *fsdev = container_of(sb, struct fs_device_d, sb);
 	struct inode *inode;
 	struct file_priv *priv;
-	int filesize;
+	loff_t filesize;
 
 	priv = tftp_do_open(&fsdev->dev, O_RDONLY, dentry);
 	if (IS_ERR(priv))
-- 
2.19.1


-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: Transferring files over tftp

[Teresa Remmet]

Hello Sascha,

Am Freitag, den 16.11.2018, 08:32 +0100 schrieb Sascha Hauer:
> Hi Teresa,
> 
> On Thu, Nov 15, 2018 at 04:24:18PM +0100, Teresa Remmet wrote:
> > 
> > Hello,
> > 
> > I am facing tftp issues when trying to transfer files > 2GB. This
> > ends up in a crash:
> > 
> > 
> > 
...
> > 
> > Causing this seems to be commit:
> > 
> > commit e3e55fa3aa698bb92fa4362a36e102749c13c124
> > Author: Sascha Hauer <s.hauer@pengutronix.de>
> > Date:   Thu Aug 30 12:27:00 2018 +0200
> > 
> >     fs: tftp: improve file size handling
> >     
> >     Previously we used FILE_SIZE_STREAM unconditionally. Instead,
> > fill the
> >     inode size with a valid filesize if we have one and only if not
> > fall
> >     back to FILE_SIZE_STREAM.
> > 
> > Reverting the commit reveals a second issue. After 2GB the transfer
> > breaks up silently.
> > Maybe some overflow? Any idea where to look at this further?
> As a first step try the attached patch which makes the filesize in
> tftp
> a 64bit type so that we at least can handle the filesize numbers.

Okay, this already seem to fix the problem. Transferring files >2GB
works here now, too. Thanks.

> 
> I tried transferring a file > 2GiB here and it seemed to work,
> although
> I don't have a board with that amount of memory, so I had to write to
> /dev/null. I don't know if that makes a difference.
> 
> Also you probably begin to hit the limits of the tftp servers. I
> realized that our tftp server (atftpd) only reports the filesize for
> files smaller than 2GiB, with bigger files it only reports '0'.

Okay I use tftpd-hpa where files >2GB work.

> 
> I think other tftp servers begin to make problems when the 16bit
> block
> counter overflows, so with working 2GiB files you are lucky already
> ;)

man page says that the tftp server also support rollover for such a
case. But this is non standard. And transferring big files with tftp is
not much fun anyway.  

Teresa

> 
> Sascha
> 
> --------------------------------------8<-----------------------------
> -----
> 
> From bf3de3391c958c9c44fcfa54b2c1a6a29e68a5a4 Mon Sep 17 00:00:00
> 2001
> From: Sascha Hauer <s.hauer@pengutronix.de>
> Date: Fri, 16 Nov 2018 08:24:06 +0100
> Subject: [PATCH] fs: tftp: Make filesize a 64bit type
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> ---
>  fs/tftp.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/tftp.c b/fs/tftp.c
> index 0d9ee6effd..8131ab27b3 100644
> --- a/fs/tftp.c
> +++ b/fs/tftp.c
> @@ -80,7 +80,7 @@ struct file_priv {
>  	int state;
>  	int err;
>  	char *filename;
> -	int filesize;
> +	loff_t filesize;
>  	uint64_t resend_timeout;
>  	uint64_t progress_timeout;
>  	struct kfifo *fifo;
> @@ -136,7 +136,7 @@ static int tftp_send(struct file_priv *priv)
>  				"timeout%c"
>  				"%d%c"
>  				"tsize%c"
> -				"%d%c"
> +				"%lld%c"
>  				"blksize%c"
>  				"1432",
>  				priv->filename + 1, 0,
> @@ -235,7 +235,7 @@ static void tftp_parse_oack(struct file_priv
> *priv, unsigned char *pkt, int len)
>  		if (val > s + len)
>  			return;
>  		if (!strcmp(opt, "tsize"))
> -			priv->filesize = simple_strtoul(val, NULL,
> 10);
> +			priv->filesize = simple_strtoull(val, NULL,
> 10);
>  		if (!strcmp(opt, "blksize"))
>  			priv->blocksize = simple_strtoul(val, NULL,
> 10);
>  		pr_debug("OACK opt: %s val: %s\n", opt, val);
> @@ -658,7 +658,7 @@ static struct dentry *tftp_lookup(struct inode
> *dir, struct dentry *dentry,
>  	struct fs_device_d *fsdev = container_of(sb, struct
> fs_device_d, sb);
>  	struct inode *inode;
>  	struct file_priv *priv;
> -	int filesize;
> +	loff_t filesize;
>  
>  	priv = tftp_do_open(&fsdev->dev, O_RDONLY, dentry);
>  	if (IS_ERR(priv))
> -- 
> 2.19.1
> 
> 

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: Transferring files over tftp

[Teresa Remmet]

Hello Sascha,

Am Freitag, den 16.11.2018, 08:32 +0100 schrieb Sascha Hauer:

...

> 
> --------------------------------------8<-----------------------------
> -----
> 
> From bf3de3391c958c9c44fcfa54b2c1a6a29e68a5a4 Mon Sep 17 00:00:00
> 2001
> From: Sascha Hauer <s.hauer@pengutronix.de>
> Date: Fri, 16 Nov 2018 08:24:06 +0100
> Subject: [PATCH] fs: tftp: Make filesize a 64bit type
> 
> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

I miss this patch on next. As it fixes my problem. Is there any reason
why it is not applied?

Thanks,
Teresa

> ---
>  fs/tftp.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/tftp.c b/fs/tftp.c
> index 0d9ee6effd..8131ab27b3 100644
> --- a/fs/tftp.c
> +++ b/fs/tftp.c
> @@ -80,7 +80,7 @@ struct file_priv {
>  	int state;
>  	int err;
>  	char *filename;
> -	int filesize;
> +	loff_t filesize;
>  	uint64_t resend_timeout;
>  	uint64_t progress_timeout;
>  	struct kfifo *fifo;
> @@ -136,7 +136,7 @@ static int tftp_send(struct file_priv *priv)
>  				"timeout%c"
>  				"%d%c"
>  				"tsize%c"
> -				"%d%c"
> +				"%lld%c"
>  				"blksize%c"
>  				"1432",
>  				priv->filename + 1, 0,
> @@ -235,7 +235,7 @@ static void tftp_parse_oack(struct file_priv
> *priv, unsigned char *pkt, int len)
>  		if (val > s + len)
>  			return;
>  		if (!strcmp(opt, "tsize"))
> -			priv->filesize = simple_strtoul(val, NULL,
> 10);
> +			priv->filesize = simple_strtoull(val, NULL,
> 10);
>  		if (!strcmp(opt, "blksize"))
>  			priv->blocksize = simple_strtoul(val, NULL,
> 10);
>  		pr_debug("OACK opt: %s val: %s\n", opt, val);
> @@ -658,7 +658,7 @@ static struct dentry *tftp_lookup(struct inode
> *dir, struct dentry *dentry,
>  	struct fs_device_d *fsdev = container_of(sb, struct
> fs_device_d, sb);
>  	struct inode *inode;
>  	struct file_priv *priv;
> -	int filesize;
> +	loff_t filesize;
>  
>  	priv = tftp_do_open(&fsdev->dev, O_RDONLY, dentry);
>  	if (IS_ERR(priv))
> -- 
> 2.19.1
> 
> 

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: Transferring files over tftp

[Sascha Hauer]

Hi Teresa,

On Thu, Nov 29, 2018 at 09:14:09AM +0100, Teresa Remmet wrote:
> Hello Sascha,
> 
> Am Freitag, den 16.11.2018, 08:32 +0100 schrieb Sascha Hauer:
> 
> ...
> 
> > 
> > --------------------------------------8<-----------------------------
> > -----
> > 
> > From bf3de3391c958c9c44fcfa54b2c1a6a29e68a5a4 Mon Sep 17 00:00:00
> > 2001
> > From: Sascha Hauer <s.hauer@pengutronix.de>
> > Date: Fri, 16 Nov 2018 08:24:06 +0100
> > Subject: [PATCH] fs: tftp: Make filesize a 64bit type
> > 
> > Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
> 
> I miss this patch on next. As it fixes my problem. Is there any reason
> why it is not applied?

No other than me forgetting about it. Applied now ;)

Sascha

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox