1729

How can I replace a newline ("\n") with a space ("") using the sed command?

I unsuccessfully tried:

sed 's#\n# #g' file
sed 's#^$# #g' file

How do I fix it?

12
  • 47
    tr is only the right tool for the job if replace a single character for a single character, while the example above shows replace newline with a space.. So in the above example, tr could work.. But would be limiting later on.
    – Angry 84
    Dec 31, 2015 at 2:47
  • 16
    tr in the right tool for the job because the questioner wanted to replace each newline with a space, as shown in his example. The replacement of newlines is uniquely arcane for sed but easily done by tr. This is a common question. Performing regex replacements is not done by tr but by sed, which would be the right tool... for a different question.
    – Mike S
    Dec 28, 2016 at 15:01
  • 3
    "tr" can also just delete the newline ` tr -d '\n' ` however you may also like to delete returns to be more universal ` tr -d '\012\015' `.
    – anthony
    Feb 27, 2017 at 23:44
  • 4
    WARNING: "tr" acts differently with regards to a character ranges between Linux and older Solaris machines (EG sol5.8). EG: ` tr -d 'a-z' ` and ` tr -d '[a-z]' `. For that I recommend you use "sed" which doesn't have that difference.
    – anthony
    Feb 27, 2017 at 23:45
  • 2
    @MikeS Thanks for the answer. Follow tr '\012' ' ' with an echo. Otherwise the last linefeed in the file is deleted, too. tr '\012' ' ' < filename; echodoes the trick. Dec 28, 2017 at 23:30

43 Answers 43

2030

sed is intended to be used on line-based input. Although it can do what you need.


A better option here is to use the tr command as follows:

tr '\n' ' ' < input_filename

or remove the newline characters entirely:

tr -d '\n' < input.txt > output.txt

or if you have the GNU version (with its long options)

tr --delete '\n' < input.txt > output.txt
8
  • 119
    Sed is line-based therefore it is hard for it to grasp newlines. Aug 9, 2009 at 19:22
  • 3
    Alexander: Does "stream editor" mean line-based? Perhaps, the name is confusing. Aug 9, 2009 at 19:26
  • 228
    sed works on a "stream" of input, but it comprehends it in newline delimited chunks. It is a unix tool, which means it does one thing very well. The one thing is "work on a file line-wise". Making it do something else will be hard, and risks being buggy. The moral of the story is: choose the right tool. A great many of your questions seem to take the form "How can I make this tool do something it was never meant to do?" Those questions are interesting, but if they come up in the course of solving a real problem, you're probably doing it wrong. Aug 9, 2009 at 19:39
  • 1
    GNU sed supports changing the "record" separator to null byte instead of newline. Jan 6, 2021 at 2:37
  • 7
    tr only works with one character strings. You can't replace all new lines with a string that is multiple characters long.
    – Flimm
    Jan 6, 2022 at 13:45
1790

Use this solution with GNU sed:

sed ':a;N;$!ba;s/\n/ /g' file

This will read the whole file in a loop (':a;N;$!ba), then replaces the newline(s) with a space (s/\n/ /g). Additional substitutions can be simply appended if needed.

Explanation:

  1. sed starts by reading the first line excluding the newline into the pattern space.
  2. Create a label via :a.
  3. Append a newline and next line to the pattern space via N.
  4. If we are before the last line, branch to the created label $!ba ($! means not to do it on the last line. This is necessary to avoid executing N again, which would terminate the script if there is no more input!).
  5. Finally the substitution replaces every newline with a space on the pattern space (which is the whole file).

Here is cross-platform compatible syntax which works with BSD and OS X's sed (as per @Benjie comment):

sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g' file

As you can see, using sed for this otherwise simple problem is problematic. For a simpler and adequate solution see this answer.

11
  • 134
    You can run this cross-platform (i.e. on Mac OS X) by separately executing the commands rather than separating with semi-colons: sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g'
    – Benjie
    Sep 27, 2011 at 11:16
  • 5
    It seems not to remove the last \n ? Aug 4, 2020 at 10:55
  • 5
    This is an impressive answer. I also find it ironic that Linux tools are supposed to be "do one thing well" when it seems like most Linux tools do many things, poorly
    – Andy Ray
    May 20, 2021 at 22:24
  • 6
    echo "Hello\nWorld" | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g' returns "Hello World", but echo "Hello World" | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/ /g' returns an empty string for me. I'm on MacOS Big Sur. May 25, 2021 at 17:15
  • 1
    @AndyRay: Really?
    – Thor
    Dec 2, 2021 at 14:49
616

Fast answer

sed ':a;N;$!ba;s/\n/ /g' file
  1. :a create a label 'a'
  2. N append the next line to the pattern space
  3. $! if not the last line, ba branch (go to) label 'a'
  4. s substitute, /\n/ regex for new line, / / by a space, /g global match (as many times as it can)

sed will loop through step 1 to 3 until it reach the last line, getting all lines fit in the pattern space where sed will substitute all \n characters


Alternatives

All alternatives, unlike sed will not need to reach the last line to begin the process

with bash, slow

while read line; do printf "%s" "$line "; done < file

with perl, sed-like speed

perl -p -e 's/\n/ /' file

with tr, faster than sed, can replace by one character only

tr '\n' ' ' < file

with paste, tr-like speed, can replace by one character only

paste -s -d ' ' file

with awk, tr-like speed

awk 1 ORS=' ' file

Other alternative like "echo $(< file)" is slow, works only on small files and needs to process the whole file to begin the process.


Long answer from the sed FAQ 5.10

5.10. Why can't I match or delete a newline using the \n escape
sequence? Why can't I match 2 or more lines using \n?

The \n will never match the newline at the end-of-line because the
newline is always stripped off before the line is placed into the
pattern space. To get 2 or more lines into the pattern space, use
the 'N' command or something similar (such as 'H;...;g;').

Sed works like this: sed reads one line at a time, chops off the
terminating newline, puts what is left into the pattern space where
the sed script can address or change it, and when the pattern space
is printed, appends a newline to stdout (or to a file). If the
pattern space is entirely or partially deleted with 'd' or 'D', the
newline is not added in such cases. Thus, scripts like

  sed 's/\n//' file       # to delete newlines from each line             
  sed 's/\n/foo\n/' file  # to add a word to the end of each line         

will NEVER work, because the trailing newline is removed before
the line is put into the pattern space. To perform the above tasks,
use one of these scripts instead:

  tr -d '\n' < file              # use tr to delete newlines              
  sed ':a;N;$!ba;s/\n//g' file   # GNU sed to delete newlines             
  sed 's/$/ foo/' file           # add "foo" to end of each line          

Since versions of sed other than GNU sed have limits to the size of
the pattern buffer, the Unix 'tr' utility is to be preferred here.
If the last line of the file contains a newline, GNU sed will add
that newline to the output but delete all others, whereas tr will
delete all newlines.

To match a block of two or more lines, there are 3 basic choices:
(1) use the 'N' command to add the Next line to the pattern space;
(2) use the 'H' command at least twice to append the current line
to the Hold space, and then retrieve the lines from the hold space
with x, g, or G; or (3) use address ranges (see section 3.3, above)
to match lines between two specified addresses.

Choices (1) and (2) will put an \n into the pattern space, where it
can be addressed as desired ('s/ABC\nXYZ/alphabet/g'). One example
of using 'N' to delete a block of lines appears in section 4.13
("How do I delete a block of specific consecutive lines?"). This
example can be modified by changing the delete command to something
else, like 'p' (print), 'i' (insert), 'c' (change), 'a' (append),
or 's' (substitute).

Choice (3) will not put an \n into the pattern space, but it does
match a block of consecutive lines, so it may be that you don't
even need the \n to find what you're looking for. Since GNU sed
version 3.02.80 now supports this syntax:

  sed '/start/,+4d'  # to delete "start" plus the next 4 lines,           

in addition to the traditional '/from here/,/to there/{...}' range
addresses, it may be possible to avoid the use of \n entirely.

12
  • 13
    tr was a great idea, and your overall coverage makes for a top-quality answer. Jan 6, 2013 at 17:57
  • 3
    +1 for using (standard utility) paste... and all the others!
    – Totor
    Mar 15, 2013 at 11:10
  • 1
    @elgalu try this unix.stackexchange.com/questions/4527/…
    – hdorio
    Dec 11, 2013 at 9:36
  • 4
    The best part about this answer is that the "long answer" explains exactly how and why the command works.
    – pdwalker
    May 2, 2014 at 11:20
  • 5
    This may be the most helpful of the thousands of answers I've read on stackexchange. I need to match multiple characters across lines. No previous sed examples covered multi-lines and tr can't handle multiple character matching. Perl looks good, but isn't working as I expect. I'd vote this answer up several times if I could.
    – mightypile
    Jun 29, 2015 at 23:37
268

GNU sed has an option, -z, for null-separated records (lines). You can just call:

sed -z 's/\n/ /g'
9
  • 6
    Even if the input does contain nulls, they will be preserved (as record delimiters). Feb 16, 2016 at 17:42
  • 8
    Won't this load the whole input if there're no nulls? In this case processing a multi-gigabyte file may be crashy.
    – Ruslan
    Mar 4, 2016 at 9:43
  • 6
    @Ruslan, yes it loads the whole input. This solution is not a good idea for multi-gigabyte files.
    – JJoao
    Mar 4, 2016 at 9:50
  • 25
    This is seriously the best answer. The other expressions are too contorted to remember. @JJoao You can use it with -u, --unbuffered. The man mage states: "load minimal amounts of data from the input files and flush the output buffers more often".
    – not2qubit
    Feb 21, 2018 at 21:10
  • 3
    The little used y command does the job too sed -z 'y/\n/ /' file.
    – potong
    Jun 3, 2021 at 12:10
254

A shorter awk alternative:

awk 1 ORS=' '

Explanation

An awk program is built up of rules which consist of conditional code-blocks, i.e.:

condition { code-block }

If the code-block is omitted, the default is used: { print $0 }. Thus, the 1 is interpreted as a true condition and print $0 is executed for each line.

When awk reads the input it splits it into records based on the value of RS (Record Separator), which by default is a newline, thus awk will by default parse the input line-wise. The splitting also involves stripping off RS from the input record.

Now, when printing a record, ORS (Output Record Separator) is appended to it, default is again a newline. So by changing ORS to a space all newlines are changed to spaces.

8
  • 6
    I like a lot this simple solution, which is much more readable, than others Jul 30, 2013 at 13:29
  • 11
    If it makes more sense, this could effectively be written as: awk 'BEGIN { ORS=" " } { print $0 } END { print "\n"} ' file.txt (adding an ending newline just to illustrate begin/end); the "1" evaluates to true (process the line) and print (print the line). A conditional could also be added to this expression, e.g., only working on lines matching a pattern: awk 'BEGIN { ORS=" " } /pattern/ { print $0 } END { print "\n"} '
    – michael
    Sep 19, 2014 at 1:22
  • 2
    You can do it more simle: code awk 'ORS=" "' file.txt code
    – Udi
    Mar 28, 2018 at 15:52
  • 1
    Why does the ORS=' ' work at the end? Is it part of the program or being interpreted as an option? I would have thought you'd need to put it in a BEGIN block or as an option before the program like awk -v ORS=' ' 1. I can't figure out exactly how awk is parsing awk 1 ORS=' '.
    – Jonah
    May 12, 2021 at 14:08
  • 2
    @Jonah: this is an alternate way of setting variables, see e.g. the GNU awk manual
    – Thor
    May 13, 2021 at 15:36
105

The Perl version works the way you expected.

perl -i -p -e 's/\n//' file

As pointed out in the comments, it's worth noting that this edits in place. -i.bak will give you a backup of the original file before the replacement in case your regular expression isn't as smart as you thought.

8
  • 29
    Please at least mention that -i without a suffix makes no backup. -i.bak protects you from an easy, ugly mistake (say, forgetting to type -p and zeroing out the file).
    – Telemachus
    Aug 9, 2009 at 20:31
  • 7
    @Telemachus: It's a fair point, but it can be argued either way. The main reason I didn't mention it is that the sed example in the OP's question doesn't make backups, so it seems superfluous here. The other reason is that I've never actually used the backup functionality (I find automatic backups annoying, actually), so I always forget it's there. The third reason is it makes my command line four characters longer. For better or worse (probably worse), I'm a compulsive minimalist; I just prefer brevity. I realise you don't agree. I will try my best to remember to warn about backups in future. Aug 9, 2009 at 20:53
  • 7
    @Ire_and_curses: Actually, you just made a damn good argument for ignoring me. That is, you have reasons for your choices, and whether or not I agree with the choices, I certainly respect that. I'm not sure entirely why, but I've been on a tear about this particular thing lately (the -i flag in Perl without a suffix). I'm sure I'll find something else to obsess about soon enough. :)
    – Telemachus
    Aug 9, 2009 at 21:36
  • It's really unfortunate that this doesn't work with stdin by specifying - for filename. Is there a way to do that? That's my go-to way to not worry about modifying a file is using a pipeline that starts with cat.
    – Steven Lu
    Apr 10, 2019 at 2:57
  • @StevenLu Perl will read from STDIN by default if no filenames are provided. So you could do e.g. perl -i -p -e 's/\n//' < infile > outfile Apr 10, 2019 at 15:47
52

Who needs sed? Here is the bash way:

cat test.txt |  while read line; do echo -n "$line "; done
14
  • 4
    Upvote, I normally used the top answer, but when piping /dev/urandom through it, sed won't print until EOF, and ^C is no EOF. This solution prints every time it sees a newline. Exactly what I needed! Thanks! Feb 9, 2011 at 17:34
  • 1
    then why not: echo -n `cat days.txt` From this post
    – Tony
    Sep 26, 2012 at 23:51
  • 10
    @Tony because backticks are deprecated and the cat is redundant ;-) Use: echo $(<days.txt)
    – seumasmac
    Feb 7, 2013 at 16:03
  • 13
    Without even using cat: while read line; do echo -n "$line "; done < test.txt. Might be useful if a sub-shell is a problem. Jan 20, 2014 at 17:42
  • 7
    echo $(<file) squeezes all whitespace to a single space, not just newlines: this goes beyond what the OP is asking. Mar 18, 2015 at 18:46
31

In order to replace all newlines with spaces using awk, without reading the whole file into memory:

awk '{printf "%s ", $0}' inputfile

If you want a final newline:

awk '{printf "%s ", $0} END {printf "\n"}' inputfile

You can use a character other than space:

awk '{printf "%s|", $0} END {printf "\n"}' inputfile
1
  • 2
    END{ print ""} is a shorter alternative for a trailing newline.
    – user8017719
    Dec 15, 2019 at 23:41
28
tr '\n' ' ' 

is the command.

Simple and easy to use.

1
  • 21
    or simply tr -d '\n' if you don't want to add a space
    – spuder
    Oct 15, 2014 at 23:26
19

Three things.

  1. tr (or cat, etc.) is absolutely not needed. (GNU) sed and (GNU) awk, when combined, can do 99.9% of any text processing you need.

  2. stream != line based. ed is a line-based editor. sed is not. See sed lecture for more information on the difference. Most people confuse sed to be line-based because it is, by default, not very greedy in its pattern matching for SIMPLE matches - for instance, when doing pattern searching and replacing by one or two characters, it by default only replaces on the first match it finds (unless specified otherwise by the global command). There would not even be a global command if it were line-based rather than STREAM-based, because it would evaluate only lines at a time. Try running ed; you'll notice the difference. ed is pretty useful if you want to iterate over specific lines (such as in a for-loop), but most of the times you'll just want sed.

  3. That being said,

    sed -e '{:q;N;s/\n/ /g;t q}' file
    

    works just fine in GNU sed version 4.2.1. The above command will replace all newlines with spaces. It's ugly and a bit cumbersome to type in, but it works just fine. The {}'s can be left out, as they're only included for sanity reasons.

4
  • 5
    As a person who only knows enough sed to do basic stuff, I have to say it's more than about what you can do with sed but rather how easy it is to understand what is going on. I have a very hard time working with sed so I would prefer a simpler command when I can use it.
    – Nate
    Mar 4, 2014 at 19:15
  • Using t q as conditional jump this works with a pattern like s/\n / / (to join all lines which begin with a space) without reading the whole file into memory. Handy when transforming multi megabyte files.
    – textshell
    Jun 26, 2015 at 11:17
  • The article you've linked does not reflect what you are saying
    – hek2mgl
    Mar 29, 2016 at 22:33
  • 3
    This is almost 800 times slower than the accepted answer on large input. This is due to running substitute for every line on increasingly larger input.
    – Thor
    Nov 3, 2016 at 20:17
18
cat file | xargs

for the sake of completeness

3
  • 3
    I'm a little rusty on bash, but isn't the cat unneeded here? Would it be better to xargs < file? Nov 12, 2021 at 14:13
  • 1
    you're right and it's shorter, but i'm just used to build pipe chains
    – nefuson
    Nov 12, 2021 at 19:53
  • Wow, this one is great but what if instead of a space you want to replace new lines with "\n"? Aug 9, 2022 at 15:08
16

Why didn't I find a simple solution with awk?

awk '{printf $0}' file

printf will print the every line without newlines, if you want to separate the original lines with a space or other:

awk '{printf $0 " "}' file
2
  • echo "1\n2\n3" | awk '{printf $0}', this works for me. @edi9999
    – Itachi
    Mar 19, 2020 at 9:39
  • this was the only approach that worked for me within git bash for windows
    – Plato
    May 15, 2020 at 0:00
15

The answer with the :a label ...

How can I replace a newline (\n) using sed?

... does not work in freebsd 7.2 on the command line:

( echo foo ; echo bar ) | sed ':a;N;$!ba;s/\n/ /g'
sed: 1: ":a;N;$!ba;s/\n/ /g": unused label 'a;N;$!ba;s/\n/ /g'
foo
bar

But does if you put the sed script in a file or use -e to "build" the sed script...

> (echo foo; echo bar) | sed -e :a -e N -e '$!ba' -e 's/\n/ /g'
foo bar

or ...

> cat > x.sed << eof
:a
N
$!ba
s/\n/ /g
eof

> (echo foo; echo bar) | sed -f x.sed
foo bar

Maybe the sed in OS X is similar.

1
  • The series of -e arguments worked for me on windows using MKS! Thanks!
    – JamesG
    Sep 7, 2011 at 19:26
15

Easy-to-understand Solution

I had this problem. The kicker was that I needed the solution to work on BSD's (Mac OS X) and GNU's (Linux and Cygwin) sed and tr:

$ echo 'foo
bar
baz


foo2
bar2
baz2' \
| tr '\n' '\000' \
| sed 's:\x00\x00.*:\n:g' \
| tr '\000' '\n'

Output:

foo
bar
baz

(has trailing newline)

It works on Linux, OS X, and BSD - even without UTF-8 support or with a crappy terminal.

  1. Use tr to swap the newline with another character.

    NULL (\000 or \x00) is nice because it doesn't need UTF-8 support and it's not likely to be used.

  2. Use sed to match the NULL

  3. Use tr to swap back extra newlines if you need them

1
  • 1
    A subtle note on nomenclature: the character \000 is commonly referred to as NUL (one L), and NULL is generally used when talking about a zero-pointer (in C/C++).
    – sqweek
    Feb 29, 2016 at 8:59
14

You can use xargs:

seq 10 | xargs

or

seq 10 | xargs echo -n
1
  • Work for me: xargs < file.txt
    – Udi
    Mar 28, 2018 at 16:31
13

If you are unfortunate enough to have to deal with Windows line endings, you need to remove the \r and the \n:

tr '\r\n' ' ' < $input > $output
3
  • This replaces [ with a space, and \r with a space, and \n with a space, and ] with a space. tr -d '\r\n' <file would remove any \r or \n characters, but that is also not what is being asked. tr -d '\r' <file will remove any \r characters (regardless of whether they are adjacent to \n) which is probably closer to being useful as well as quite possibly correct for the OP's need (still assuming your tr understands this backslash notation).
    – tripleee
    Dec 29, 2019 at 10:39
  • Thanks, fixed it. just don't put [], and tr does respect \n & \r as new line and returns. are there systems where tr doesn't? Jul 16, 2020 at 1:08
  • They ase oretty ubiquitous these days, but I think I can remember systems where they didn't work (dinosaurs like HP-UX and AIX and Irix maybe?)
    – tripleee
    Jul 16, 2020 at 5:48
9

I'm not an expert, but I guess in sed you'd first need to append the next line into the pattern space, bij using "N". From the section "Multiline Pattern Space" in "Advanced sed Commands" of the book sed & awk (Dale Dougherty and Arnold Robbins; O'Reilly 1997; page 107 in the preview):

The multiline Next (N) command creates a multiline pattern space by reading a new line of input and appending it to the contents of the pattern space. The original contents of pattern space and the new input line are separated by a newline. The embedded newline character can be matched in patterns by the escape sequence "\n". In a multiline pattern space, the metacharacter "^" matches the very first character of the pattern space, and not the character(s) following any embedded newline(s). Similarly, "$" matches only the final newline in the pattern space, and not any embedded newline(s). After the Next command is executed, control is then passed to subsequent commands in the script.

From man sed:

[2addr]N

Append the next line of input to the pattern space, using an embedded newline character to separate the appended material from the original contents. Note that the current line number changes.

I've used this to search (multiple) badly formatted log files, in which the search string may be found on an "orphaned" next line.

0
7

In response to the "tr" solution above, on Windows (probably using the Gnuwin32 version of tr), the proposed solution:

tr '\n' ' ' < input

was not working for me, it'd either error or actually replace the \n w/ '' for some reason.

Using another feature of tr, the "delete" option -d did work though:

tr -d '\n' < input

or '\r\n' instead of '\n'

3
  • 3
    On Windows, you probably need to use tr "\n" " " < input. The Windows shell (cmd.exe) doesn't treat the apostrophe as a quoting character. Aug 24, 2011 at 14:25
  • No, in Windows 10 Ubuntu subsystem, you need to use tr "\n\r" " " < input.txt > output.txt Sep 29, 2017 at 3:55
  • This works on Windows 10 using Gnuwin32: cat SourceFile.txt | tr --delete '\r\n' > OutputFile.txt . Or, instead of Gnuwin32, use Gow (Gnu on Windows), github.com/bmatzelle/gow/wiki Jun 13, 2018 at 23:15
7

I used a hybrid approach to get around the newline thing by using tr to replace newlines with tabs, then replacing tabs with whatever I want. In this case, "
" since I'm trying to generate HTML breaks.

echo -e "a\nb\nc\n" |tr '\n' '\t' | sed 's/\t/ <br> /g'`
7

You can also use this method:

sed 'x;G;1!h;s/\n/ /g;$!d'

Explanation

x   - which is used to exchange the data from both space (pattern and hold).
G   - which is used to append the data from hold space to pattern space.
h   - which is used to copy the pattern space to hold space.
1!h - During first line won't copy pattern space to hold space due to \n is
      available in pattern space.
$!d - Clear the pattern space every time before getting the next line until the
      the last line.

Flow

When the first line get from the input, an exchange is made, so 1 goes to hold space and \n comes to pattern space, appending the hold space to pattern space, and a substitution is performed and deletes the pattern space.

During the second line, an exchange is made, 2 goes to hold space and 1 comes to the pattern space, G append the hold space into the pattern space, h copy the pattern to it, the substitution is made and deleted. This operation is continued until EOF is reached and prints the exact result.

2
  • 1
    However, be warned that echo 'Y' | sed 'x;G;1!h;s/\n/X/g;$!d' results in XY.
    – Spooky
    Mar 13, 2015 at 1:41
  • This does affect the last \n in the file.
    – Torben
    Oct 4, 2023 at 8:40
5

Bullet-proof solution. Binary-data-safe and POSIX-compliant, but slow.

POSIX sed requires input according to the POSIX text file and POSIX line definitions, so NULL-bytes and too long lines are not allowed and each line must end with a newline (including the last line). This makes it hard to use sed for processing arbitrary input data.

The following solution avoids sed and instead converts the input bytes to octal codes and then to bytes again, but intercepts octal code 012 (newline) and outputs the replacement string in place of it. As far as I can tell the solution is POSIX-compliant, so it should work on a wide variety of platforms.

od -A n -t o1 -v | tr ' \t' '\n\n' | grep . |
  while read x; do [ "0$x" -eq 012 ] && printf '<br>\n' || printf "\\$x"; done

POSIX reference documentation: sh, shell command language, od, tr, grep, read, [, printf.

Both read, [, and printf are built-ins in at least bash, but that is probably not guaranteed by POSIX, so on some platforms it could be that each input byte will start one or more new processes, which will slow things down. Even in bash this solution only reaches about 50 kB/s, so it's not suited for large files.

Tested on Ubuntu (bash, dash, and busybox), FreeBSD, and OpenBSD.

5

In some situations maybe you can change RS to some other string or character. This way, \n is available for sub/gsub:

$ gawk 'BEGIN {RS="dn" } {gsub("\n"," ") ;print $0 }' file

The power of shell scripting is that if you do not know how to do it in one way you can do it in another way. And many times you have more things to take into account than make a complex solution on a simple problem.

Regarding the thing that gawk is slow... and reads the file into memory, I do not know this, but to me gawk seems to work with one line at the time and is very very fast (not that fast as some of the others, but the time to write and test also counts).

I process MB and even GB of data, and the only limit I found is line size.

0
5

Finds and replaces using allowing \n

sed -ie -z 's/Marker\n/# Marker Comment\nMarker\n/g' myfile.txt

Marker

Becomes

# Marker Comment

Marker

0
4

You could use xargs — it will replace \n with a space by default.

However, it would have problems if your input has any case of an unterminated quote, e.g. if the quote signs on a given line don't match.

1
  • xargs also handles the last line nicely: Aug 22, 2015 at 21:37
3

On Mac OS X (using FreeBSD sed):

# replace each newline with a space
printf "a\nb\nc\nd\ne\nf" | sed -E -e :a -e '$!N; s/\n/ /g; ta'
printf "a\nb\nc\nd\ne\nf" | sed -E -e :a -e '$!N; s/\n/ /g' -e ta
3

To remove empty lines:

sed -n "s/^$//;t;p;"
1
  • This is for GNU Sed. In normal Sed, this gives sed: 1: "s/^$//;t;p;": undefined label ';p;'. Jun 8, 2015 at 20:29
3

Using Awk:

awk "BEGIN { o=\"\" }  { o=o \" \" \$0 }  END { print o; }"
3
  • 2
    You don't need to escape the quotation marks and dollar sign if you change the outer ones to single quotes. The letter "o" is usually considered a bad choice as a variable name since it can be confused with the digit "0". You also don't need to initialize your variable, it defaults to a null string. However, if you don't want an extraneous leading space: awk '{s = s sp $0; sp = " "} END {print s}'. However, see my answer for a way to use awk without reading the whole file into memory. Mar 30, 2012 at 6:34
  • Please check out Thor's answer instead. It is way more efficient, readable and just better by all means to compared this approach (even though this would work)!
    – mschilli
    Sep 2, 2013 at 17:03
  • Dude, I get it. No need to rub it in my face :-) Thor's answer is way above on the page anyway (which is right), so what do you care?
    – kralyk
    Sep 3, 2013 at 13:28
3

A solution I particularly like is to append all the file in the hold space and replace all newlines at the end of file:

$ (echo foo; echo bar) | sed -n 'H;${x;s/\n//g;p;}'
foobar

However, someone said me the hold space can be finite in some sed implementations.

1
  • 1
    the replacement with an empty string in your answer conceals the fact that always using H to append to the hold space means that the hold space will start with a newline. To avoid this, you need to use 1h;2,$H;${x;s/\n/x/g;p}
    – Jeff
    Jul 9, 2016 at 6:34
3

Replace newlines with any string, and replace the last newline too

The pure tr solutions can only replace with a single character, and the pure sed solutions don't replace the last newline of the input. The following solution fixes these problems, and seems to be safe for binary data (even with a UTF-8 locale):

printf '1\n2\n3\n' |
  sed 's/%/%p/g;s/@/%a/g' | tr '\n' @ | sed 's/@/<br>/g;s/%a/@/g;s/%p/%/g'

Result:

1<br>2<br>3<br>
3
  • This is bad because it will produce unwanted output on any input containing @
    – Steven Lu
    Jun 22, 2013 at 17:38
  • @StevenLu: No, @ in the input is OK. It gets escaped to %a and back again. The solution might not be completely POSIX compliant, though (NULL-bytes not allowed so not good for binary data, and all lines must end with newline so the tr output is not really valid). Jun 27, 2013 at 0:48
  • Ah. I see you've fixed it up. Kinda convoluted for what should be a simple operation, but good work.
    – Steven Lu
    Jun 27, 2013 at 2:48
3

It is sed that introduces the new-lines after "normal" substitution. First, it trims the new-line char, then it processes according to your instructions, then it introduces a new-line.

Using sed you can replace "the end" of a line (not the new-line char) after being trimmed, with a string of your choice, for each input line; but, sed will output different lines. For example, suppose you wanted to replace the "end of line" with "===" (more general than a replacing with a single space):

PROMPT~$ cat <<EOF |sed 's/$/===/g'
first line
second line
3rd line
EOF

first line===
second line===
3rd line===
PROMPT~$

To replace the new-line char with the string, you can, inefficiently though, use tr , as pointed before, to replace the newline-chars with a "special char" and then use sed to replace that special char with the string you want.

For example:

PROMPT~$ cat <<EOF | tr '\n' $'\x01'|sed -e 's/\x01/===/g'
first line
second line
3rd line
EOF

first line===second line===3rd line===PROMPT~$

Not the answer you're looking for? Browse other questions tagged or ask your own question.