How to show full column content in a Spark Dataframe?

results.show(20, false) will not truncate. Check the source

20 is the default number of rows displayed when show() is called without any arguments.

---

If you put results.show(false) , results will not be truncated

---

Below code would help to view all rows without truncation in each column

df.show(df.count(), False)

---

The other solutions are good. If these are your goals:

No truncation of columns, No loss of rows, Fast and Efficient

These two lines are useful ...

    df.persist
    df.show(df.count, false) // in Scala or 'False' in Python

By persisting, the 2 executor actions, count and show, are faster & more efficient when using persist or cache to maintain the interim underlying dataframe structure within the executors. See more about persist and cache.

---

results.show(20, False) or results.show(20, false) depending on whether you are running it on Java/Scala/Python

---

In Pyspark we can use

df.show(truncate=False) this will display the full content of the columns without truncation.

df.show(5,truncate=False) this will display the full content of the first five rows.

---

The following answer applies to a Spark Streaming application.

By setting the "truncate" option to false, you can tell the output sink to display the full column.

val query = out.writeStream
          .outputMode(OutputMode.Update())
          .format("console")
          .option("truncate", false)
          .trigger(Trigger.ProcessingTime("5 seconds"))
          .start()

---

In c# Option("truncate", false) does not truncate data in the output.

StreamingQuery query = spark
                    .Sql("SELECT * FROM Messages")
                    .WriteStream()
                    .OutputMode("append")
                    .Format("console")
                    .Option("truncate", false)
                    .Start();

---

results.show(false) will show you the full column content.

Show method by default limit to 20, and adding a number before false will show more rows.

---

Within Databricks you can visualize the dataframe in a tabular format. With the command:

display(results)

It will look like

https://i.stack.imgur.com/g0Svi.png

---

results.show(20,false) did the trick for me in Scala.

---

Try df.show(20,False)

Notice that if you do not specify the number of rows you want to show, it will show 20 rows but will execute all your dataframe which will take more time !

---

try this command :

df.show(df.count())

---

Tried this in pyspark

df.show(truncate=0)

---

In Spark Pythonic way, remember:

if you have to display data from a dataframe, use show(truncate=False) method.

else if you have to display data from a Stream dataframe view (Structured Streaming), use the writeStream.format("console").option("truncate", False).start() methods with option.

Hope it could helps someone.

---

I use the plugin Chrome extension works pretty well:

[https://userstyles.org/styles/157357/jupyter-notebook-wide][1]

---

Try this in scala:

df.show(df.count.toInt, false)

The show method accepts an integer and a Boolean value but df.count returns Long...so type casting is required

---

PYSPARK

In the below code, df is the name of dataframe. 1st parameter is to show all rows in the dataframe dynamically rather than hardcoding a numeric value. The 2nd parameter will take care of displaying full column contents since the value is set as False.

df.show(df.count(),False)

https://i.stack.imgur.com/Lu6Dv.png

SCALA

In the below code, df is the name of dataframe. 1st parameter is to show all rows in the dataframe dynamically rather than hardcoding a numeric value. The 2nd parameter will take care of displaying full column contents since the value is set as false.

df.show(df.count().toInt,false)

https://i.stack.imgur.com/Bv9Kw.png

---