Subject: Re: Create files with specific sizes? - msg#00057

Previous Message by Date: click to view message preview

RE: zero copy issue while receiving the data (counter part of sen dfil e)

On Fri, 2004-12-17 at 21:54 +0530, Rajat Jain, Noida wrote: > > Hi, > > Thanks for the reply. > > Actually I am developing a loadable kernel module. I agree that at the bare > minimum, I need to copy from the NIC's device buffer to kernel's allocated > sk_buff (socket buffer). What I want is to avoid FURTHER coying of data from > the sk_buffs to the buffers allocated by the module. Looks like you have two options: a) pre-fill and use "struct iovec" with sock_recvmsg() b) intercept socket's receive callback with tcp_read_sock() and use skb_copy_bits() to copy data from skb to your destination buffer. Regards, Dima > > And hence I expected to pass the address of a buffer pointer to > tcp_read_sock(). And I expected this function to set it to socket buffer. > Any pointers on the functionality of tcp_read_sock()?? > > Rajat > > > -----Original Message----- > From: Dmitry Yusupov [mailto:dima@xxxxxxxx] > Sent: Friday, December 17, 2004 7:07 AM > To: Rajat Jain, Noida > Cc: linux-net@xxxxxxxxxxxxxxx; Sanjay Kumar, Noida; Deepak Kumar Gupta, > Noida > Subject: Re: zero copy issue while receiving the data (counter part of > sendfil e) > > Hi Rajat, > > I was using this function some times back... It's been working for me just > fine. Also kernel's RPC (see xprt* files) uses it. So you might want to take > a look. > > In general, it is not possible to fully avoid copying. You need at least > copy data from NIC's skb to the destination. It might be user buffer or > kernel buffer(depends on application). > > Regards, > Dmitry > > > On Thu, 2004-12-16 at 19:38 +0530, Rajat Jain, Noida wrote: > > > > Hi, > > > > I'm experimenting on stock kernel 2.6.8 > > > > I was looking for an interface that could directly receive data from a > > network socket, WITHOUT coying from kernel space to user space. (Like > > for sending data, "sendfile" provides to send data to network socket > > without copying it to kernel space). I came across tcp_read_sock() > > interface in net/ipv4/tcp.c. > > > > Has anybody tried tcp_read_sock()?? Is there any known issue with it > > ?? If somebody has some idea, I would appreciate if you can share. > > > > I might be wrong, but what I perceive is that I will pass a pointer to > > this function. And when the function returns, I expect it to be set to > > the kernel buffer (corresponding to socket). > > > > 1) To fulfill this objective, I expect to pass a pointer to pointer & > > only then it can be done. (If we have to modify a pointer's value, we > > have to pass its address ... Right??). However, this function expects > > a char * buf (in read_descriptor_t argument). Any ideas ????????? > > > > 2) This code also frees the space allocated to sk_buffs etc using > > sk_eat_skb(sk, skb) and cleanup_rbuf(sk, copied) etc. But this > > function is supposed to return these locations to the calling code ... > Right??? > > > > Any pointers are more than welcome. I have provided the code for > reference. > > Please cc the reply to me as I'm not on the list. > > > > Thanks & regards, > > > > Rajat Jain > > > > ---------------------------------------------------------------------- > > - > > /* net/ipv4/tcp.c > > * This routine provides an alternative to tcp_recvmsg() for routines > > * that would like to handle copying from skbuffs directly in 'sendfile' > > * fashion. > > * Note: > > * - It is assumed that the socket was locked by the caller. > > * - The routine does not block. > > * - At present, there is no support for reading OOB data > > * or for 'peeking' the socket using this routine > > * (although both would be easy to implement). > > */ > > int tcp_read_sock(struct sock *sk, read_descriptor_t *desc, > > sk_read_actor_t recv_actor) { > > struct sk_buff *skb; > > struct tcp_opt *tp = tcp_sk(sk); > > u32 seq = tp->copied_seq; > > u32 offset; > > int copied = 0; > > > > if (sk->sk_state == TCP_LISTEN) > > return -ENOTCONN; > > while ((skb = tcp_recv_skb(sk, seq, &offset)) != NULL) { > > if (offset < skb->len) { > > size_t used, len; > > > > len = skb->len - offset; > > /* Stop reading if we hit a patch of urgent data > */ > > if (tp->urg_data) { > > u32 urg_offset = tp->urg_seq - seq; > > if (urg_offset < len) > > len = urg_offset; > > if (!len) > > break; > > } > > used = recv_actor(desc, skb, offset, len); > > if (used <= len) { > > seq += used; > > copied += used; > > offset += used; > > } > > if (offset != skb->len) > > break; > > } > > if (skb->h.th->fin) { > > sk_eat_skb(sk, skb); > > ++seq; > > break; > > } > > sk_eat_skb(sk, skb); > > if (!desc->count) > > break; > > } > > tp->copied_seq = seq; > > > > tcp_rcv_space_adjust(sk); > > > > /* Clean up data we have read: This will do ACK frames. */ > > if (copied) > > cleanup_rbuf(sk, copied); > > return copied; > > }--------------------------------------------------------------------- > > -- > > > > read_descriptor_t is defined as: > > > > /* > > * include/linux/fs.h > > */ > > typedef struct { > > size_t written; > > size_t count; > > union { > > char __user * buf; > > void *data; > > } arg; > > int error; > > } read_descriptor_t; > > ---------------------------------------------------------------------- > > - > > > > - > > To unsubscribe from this list: send the line "unsubscribe linux-net" > > in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo > > info at http://vger.kernel.org/majordomo-info.html -- Kernelnewbies: Help each other learn about the Linux kernel. Archive: http://mail.nl.linux.org/kernelnewbies/ FAQ: http://kernelnewbies.org/faq/

Next Message by Date: click to view message preview

The du and ls -l

Hi, In some cases, as have been mentioned in Advanced Programming in Unix Environment, by Richard Stevens, files with holes report different sizes with du -s and ls -l. But how does the reporting of du and ls -l actually differ? What is the fundamental difference between them?? Hoping for a help TIA -- With regards, Jagadeesh Bhaskar P - To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.linux-learn.org/faqs

Previous Message by Thread: click to view message preview

Re: Create files with specific sizes?

... I wanted a file of garbage, not zeroes. So I tried: dd if=/dev/random of=file bs=1k count=1024 ... That's exactly what it is supposed to do. Perhaps you tought /dev/random could generate random numbers much faster. So when it's entropy pool was exhausted after 4KB, you tought it stopped working. Nope I did not abort it, I just moved the mouse to generate more random numbers (with dd running in X terminal). What dd says is: 0+1024 records in 0+1024 records out But the file still is just ~4k. Maybe a bug? // Jarmo - To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.linux-learn.org/faqs

Next Message by Thread: click to view message preview

Re: Create files with specific sizes?

Simon Valiquette wrote: ... I wanted a file of garbage, not zeroes. So I tried: dd if=/dev/random of=file bs=1k count=1024 ... You have read and written 1024 partial blocks (ie. not complete). Why? I don't know. You did'nt say if the produced file is always the same size, or differents but always bigger (or smaller) than 4K? Please give the exact value(s) as it can hint at what the problem is. I have the impression that you get only 4 bytes per block readed. You can experiment with different values to try to find a patern. The size varies. Did a few test: 2843..3815..7414 bytes. Biggest with: bs=512 count=1024x2. Others with: bs=1024 count=1024 (only the smallest and biggest listed). Also, check if the produced file really is random data. It looks random alright. Try "cp /dev/urandom /tmp/file.cp" and type CTRL-C after waiting a /dev/urandom works. Its /dev/random that does not. Did: "cp /dev/random file" and it did block (as expected) until I aborted it (31299 bytes). It seems that dd gives up after a while. ... By curiosity, try this and report the exact file size produced (and check if it is random data). dd if=/dev/urandom of=/tmp/file.cp bs=1 count=1024x1024 Well it did produce: 1048576 bytes of random data. As expected. dd if=/dev/random of=/tmp/file.cp bs=1 count=1024x1024 Does block dd (as it waits for more values), as expected (got bored and aborted it, result 52381 bytes). The problem just occurs with /dev/random and blocksizes bigger than 1. Tried: bs=512 count=1024x2 - got 7414 Tried: bs=512 count=2 - got 680 ... But the file still is just ~4k. Maybe a bug? ... more. You did'nt even said which Linux distribution you are using, so Once it was SuSE 8.1 (or 8.2 I do not remember). dd is version 4.5.8. kernel is build by SuSE ( 2.4.20-4GB-athlon ). with that little information I can't help much. By the way, I can't reproduce your problem. Well nothing to loose sleep over. I was just suprised of the result. Anyway this computer is due an upgrade. Did for fun test this on another (home built) system with dd version 5.2.1. There it worked as expected, blocked until I aborted it (378 bytes). The low count is probably because I accessed that computer remotly with ssh (no mouse motion that provides seed to /dev/random). So this "problem" is probably solved by upgrading dd. Thanks for your time. // Jarmo - To unsubscribe from this list: send the line "unsubscribe linux-newbie" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.linux-learn.org/faqs