用ChatGPT解决这个技术问题 Extra ChatGPT

How do I remove repeated elements from ArrayList?

I have an ArrayList<String>, and I want to remove repeated strings from it. How can I do this?

I
Ivar

If you don't want duplicates in a Collection, you should consider why you're using a Collection that allows duplicates. The easiest way to remove repeated elements is to add the contents to a Set (which will not allow duplicates) and then add the Set back to the ArrayList: 

Set<String> set = new HashSet<>(yourList);
yourList.clear();
yourList.addAll(set);

Of course, this destroys the ordering of the elements in the ArrayList.

See also LinkedHashSet, if you wish to retain the order.
@Chetan finding all duplicates from ArrayList in O(n), its important to have correctly defined equals method on objects which you have in the list (no problem for numbers): public Set<Object> findDuplicates(List<Object> list) { Set<Object> items = new HashSet<Object>(); Set<Object> duplicates = new HashSet<Object>(); for (Object item : list) { if (items.contains(item)) { duplicates.add(item); } else { items.add(item); } } return duplicates; }
A good practice would be to define variables using the interface types List and Set (instead of implementation types ArrayList and HashSet as in your example).
You can clean this up by using new HashSet(al) instead of initializing it to empty and calling addAll.
Alternative: Set<Object> set = new HashSet<>(); yourList.removeIf(x -> !set.add(x)); The advantage is that this alternative allows you to decide what to use in the set.add(…) expression, for your particular notion of “duplicate”. It’s also independent of the list’s actual elements type. It also retains the order, regardless of whether the set maintains the order or not. Also usable with a TreeSet (e.g. with custom comparator) instead of HashSet.
I
IgorGanapolsky

Although converting the ArrayList to a HashSet effectively removes duplicates, if you need to preserve insertion order, I'd rather suggest you to use this variant 

// list is some List of Strings
Set<String> s = new LinkedHashSet<>(list);

Then, if you need to get back a List reference, you can use again the conversion constructor.

Does LinkedHashSet make any guarantees as to which of several duplicates are kept from the list? For instance, if position 1, 3, and 5 are duplicates in the original list, can we assume that this process will remove 3 and 5? Or maybe remove 1 and 3? Thanks.
@Matt: yes, it does guarantee that. The docs say: "This linked list defines the iteration ordering, which is the order in which elements were inserted into the set (insertion-order). Note that insertion order is not affected if an element is re-inserted into the set."
Very interesting. I have a different situation here. I am not trying to sort String but another object called AwardYearSource. This class has an int attribute called year. So I want to remove duplicates based on the year. i.e if there is year 2010 mentioned more than once, I want to remove that AwardYearSource object. How can I do that?
@WowBow For example you can define Wrapper object which holds AwardYearSource. And define this Wrapper objects equals method based on AwardYearSources year field. Then you can use Set with these Wrapper objects.
@WowBow or implement Comparable/Comparator
C
Community

In Java 8: 

List<String> deduped = list.stream().distinct().collect(Collectors.toList());

Please note that the hashCode-equals contract for list members should be respected for the filtering to work properly.

How do i do this for case insensitive distinct ?
@StackFlowed If you don't need to preserve the order of the list you can addAll to new TreeSet<String>(String.CASE_INSENSITIVE_ORDER). The first element added will remain in the set so if your list contains "Dog" and "dog" (in that order) the TreeSet will contain "Dog". If order must be preserved then before the line in the answer put list.replaceAll(String::toUpperCase);.
I am getting this error :incompatible types: List cannot be converted to List
This is a simple solution in general but how do you remove the duplicates from an Arraylist of int[]?
a
akhil_mittal

Suppose we have a list of String like: 

List<String> strList = new ArrayList<>(5);
// insert up to five items to list.

Then we can remove duplicate elements in multiple ways.

Prior to Java 8 

List<String> deDupStringList = new ArrayList<>(new HashSet<>(strList));

Note: If we want to maintain the insertion order then we need to use LinkedHashSet in place of HashSet

Using Guava 

List<String> deDupStringList2 = Lists.newArrayList(Sets.newHashSet(strList));

Using Java 8 

List<String> deDupStringList3 = strList.stream().distinct().collect(Collectors.toList());

Note: In case we want to collect the result in a specific list implementation e.g. LinkedList then we can modify the above example as: 

List<String> deDupStringList3 = strList.stream().distinct()
                 .collect(Collectors.toCollection(LinkedList::new));

We can use parallelStream also in the above code but it may not give expected performace benefits. Check this question for more.

Yah, When i typed my previous comments, I was in a impression that parallel streams will give better performance always. But it's a myth. I later learned that there are certain scenarios where parallel streams should be used. In this scenario parallel streams will not give any better performance. and yes parallel streams might not give desired results some cases. List<String> deDupStringList3 = stringList.stream().map(String::toLowerCase).distinct().collect(Collectors.toList()); should be the suitable solution in this case
B
Benno Richters

If you don't want duplicates, use a Set instead of a List. To convert a List to a Set you can use the following code: 

// list is some List of Strings
Set<String> s = new HashSet<String>(list);

If really necessary you can use the same construction to convert a Set back into a List.

Similarly at the bottom of the thread, I have given an answer where I am using Set for Custom Object. In a case if anyone have custom object like "Contact" or "Student" can use that answer that works fine for me.
The problem comes when you have to specifically access an element. For instance when binding an object to a list item view in Android, you are given its index. So Set cannot be used here.
How can I aproach this when the list is an object list
N
Nenad Bulatović

You can also do it this way, and preserve order: 

// delete duplicates (if any) from 'myArrayList'
myArrayList = new ArrayList<String>(new LinkedHashSet<String>(myArrayList));
I think this is the best way of removing duplicated in an ArrayList. Definitely recommended. Thank you @Nenad for the answer.
i
infoj

Java 8 streams provide a very simple way to remove duplicate elements from a list. Using the distinct method. If we have a list of cities and we want to remove duplicates from that list it can be done in a single line - 

 List<String> cityList = new ArrayList<>();
 cityList.add("Delhi");
 cityList.add("Mumbai");
 cityList.add("Bangalore");
 cityList.add("Chennai");
 cityList.add("Kolkata");
 cityList.add("Mumbai");

 cityList = cityList.stream().distinct().collect(Collectors.toList());

How to remove duplicate elements from an arraylist

P
Pritam Banerjee

Here's a way that doesn't affect your list ordering: 

ArrayList l1 = new ArrayList();
ArrayList l2 = new ArrayList();

Iterator iterator = l1.iterator();

while (iterator.hasNext()) {
    YourClass o = (YourClass) iterator.next();
    if(!l2.contains(o)) l2.add(o);
}

l1 is the original list, and l2 is the list without repeated items (Make sure YourClass has the equals method according to what you want to stand for equality)

This answer lacks two things: 1) It does not use generics, but raw types (ArrayList<T> should be used instead of ArrayList) 2) The explicit iterator creating can be avoided by using a for (T current : l1) { ... }. Even if you wanted to use an Iterator explicitly, iterador is misspelled.
And this implementation runs in quadratic time, compared to the linked hash set implementation running in linear time. (i.e. this takes 10 times longer on a list with 10 elements, 10,000 times longer on a list with 10,000 elements. JDK 6 implementation for ArrayList.contains, JDK8 impl is the same.)
P
Pritam Banerjee

this can solve the problem: 

private List<SomeClass> clearListFromDuplicateFirstName(List<SomeClass> list1) {

     Map<String, SomeClass> cleanMap = new LinkedHashMap<String, SomeClass>();
     for (int i = 0; i < list1.size(); i++) {
         cleanMap.put(list1.get(i).getFirstName(), list1.get(i));
     }
     List<SomeClass> list = new ArrayList<SomeClass>(cleanMap.values());
     return list;
}
I liked this solution better.
C
CarlJohn

It is possible to remove duplicates from arraylist without using HashSet or one more arraylist.

Try this code.. 

    ArrayList<String> lst = new ArrayList<String>();
    lst.add("ABC");
    lst.add("ABC");
    lst.add("ABCD");
    lst.add("ABCD");
    lst.add("ABCE");

    System.out.println("Duplicates List "+lst);

    Object[] st = lst.toArray();
      for (Object s : st) {
        if (lst.indexOf(s) != lst.lastIndexOf(s)) {
            lst.remove(lst.lastIndexOf(s));
         }
      }

    System.out.println("Distinct List "+lst);

Output is 

Duplicates List [ABC, ABC, ABCD, ABCD, ABCE]
Distinct List [ABC, ABCD, ABCE]
It's slow and you might get a ConcurrentModificationException.
@maaartinus Have you tried that code ?. It won't produce any exceptions.Also it is pretty fast. I tried the code before posting.
You're right, it doesn't as you iterate the array instead of the list. However, it's slow like hell. Try it with a few millions elements. Compare it to ImmutableSet.copyOf(lst).toList().
answers the question I was asked in the interview .. How to remove repeated values from an ArrayList without using Sets. Thanx
Internally, indexOf iterates the lst using a for loop.
r
randers

There is also ImmutableSet from Guava as an option (here is the documentation): 

ImmutableSet.copyOf(list);
Note that there is an ImmutableSet.asList() method, returning an ImmutableList, if you need it back as a List.
v
volley

Probably a bit overkill, but I enjoy this kind of isolated problem. :)

This code uses a temporary Set (for the uniqueness check) but removes elements directly inside the original list. Since element removal inside an ArrayList can induce a huge amount of array copying, the remove(int)-method is avoided. 

public static <T> void removeDuplicates(ArrayList<T> list) {
    int size = list.size();
    int out = 0;
    {
        final Set<T> encountered = new HashSet<T>();
        for (int in = 0; in < size; in++) {
            final T t = list.get(in);
            final boolean first = encountered.add(t);
            if (first) {
                list.set(out++, t);
            }
        }
    }
    while (out < size) {
        list.remove(--size);
    }
}

While we're at it, here's a version for LinkedList (a lot nicer!): 

public static <T> void removeDuplicates(LinkedList<T> list) {
    final Set<T> encountered = new HashSet<T>();
    for (Iterator<T> iter = list.iterator(); iter.hasNext(); ) {
        final T t = iter.next();
        final boolean first = encountered.add(t);
        if (!first) {
            iter.remove();
        }
    }
}

Use the marker interface to present a unified solution for List: 

public static <T> void removeDuplicates(List<T> list) {
    if (list instanceof RandomAccess) {
        // use first version here
    } else {
        // use other version here
    }
}

EDIT: I guess the generics-stuff doesn't really add any value here.. Oh well. :)

Why use ArrayList in parameter? Why not just List? Will that not work?
A List will absolutely work as in-parameter for the first method listed. The method is however optimized for use with a random access list such as ArrayList, so if a LinkedList is passed instead you will get poor performance. For example, setting the n:th element in a LinkedList takes O(n) time, whereas setting the n:th element in a random access list (such as ArrayList) takes O(1) time. Again, though, this is probably overkill... If you need this kind of specialized code it will hopefully be in an isolated situation.
M
Manash Ranjan Dakua 
public static void main(String[] args){
    ArrayList<Object> al = new ArrayList<Object>();
    al.add("abc");
    al.add('a');
    al.add('b');
    al.add('a');
    al.add("abc");
    al.add(10.3);
    al.add('c');
    al.add(10);
    al.add("abc");
    al.add(10);
    System.out.println("Before Duplicate Remove:"+al);
    for(int i=0;i<al.size();i++){
        for(int j=i+1;j<al.size();j++){
            if(al.get(i).equals(al.get(j))){
                al.remove(j);
                j--;
            }
        }
    }
    System.out.println("After Removing duplicate:"+al);
}
This implementation return no element in the list because of the last j--
This implementation work's very fine.there is no issue behind this and for this task i am only use one arraylist.so this answer is completely good.before giving negative feedback you shold also add testcase also so that every one can understand the result.Thanks Manash
D
Donald Raab

If you're willing to use a third-party library, you can use the method distinct() in Eclipse Collections (formerly GS Collections). 

ListIterable<Integer> integers = FastList.newListWith(1, 3, 1, 2, 2, 1);
Assert.assertEquals(
    FastList.newListWith(1, 3, 2),
    integers.distinct());

The advantage of using distinct() instead of converting to a Set and then back to a List is that distinct() preserves the order of the original List, retaining the first occurrence of each element. It's implemented by using both a Set and a List. 

MutableSet<T> seenSoFar = UnifiedSet.newSet();
int size = list.size();
for (int i = 0; i < size; i++)
{
    T item = list.get(i);
    if (seenSoFar.add(item))
    {
        targetCollection.add(item);
    }
}
return targetCollection;

If you cannot convert your original List into an Eclipse Collections type, you can use ListAdapter to get the same API. 

MutableList<Integer> distinct = ListAdapter.adapt(integers).distinct();

Note: I am a committer for Eclipse Collections.

S
Saurabh Gaddelpalliwar

If you are using model type List< T>/ArrayList< T> . Hope,it's help you.

Here is my code without using any other data structure like set or hashmap 

for (int i = 0; i < Models.size(); i++){
for (int j = i + 1; j < Models.size(); j++) {       
 if (Models.get(i).getName().equals(Models.get(j).getName())) {    
 Models.remove(j);
   j--;
  }
 }
}
R
RAS

If you want to preserve your Order then it is best to use LinkedHashSet. Because if you want to pass this List to an Insert Query by Iterating it, the order would be preserved.

Try this 

LinkedHashSet link=new LinkedHashSet();
List listOfValues=new ArrayList();
listOfValues.add(link);

This conversion will be very helpful when you want to return a List but not a Set.

M
M Kaweepatt Churcharoen

This three lines of code can remove the duplicated element from ArrayList or any collection. 

List<Entity> entities = repository.findByUserId(userId);

Set<Entity> s = new LinkedHashSet<Entity>(entities);
entities.clear();
entities.addAll(s);
G
Ghyour 
for(int a=0;a<myArray.size();a++){
        for(int b=a+1;b<myArray.size();b++){
            if(myArray.get(a).equalsIgnoreCase(myArray.get(b))){
                myArray.remove(b); 
                dups++;
                b--;
            }
        }
}
M
MantaMan

When you are filling the ArrayList, use a condition for each element. For example: 

    ArrayList< Integer > al = new ArrayList< Integer >(); 

    // fill 1 
    for ( int i = 0; i <= 5; i++ ) 
        if ( !al.contains( i ) ) 
            al.add( i ); 

    // fill 2 
    for (int i = 0; i <= 10; i++ ) 
        if ( !al.contains( i ) ) 
            al.add( i ); 

    for( Integer i: al )
    {
        System.out.print( i + " ");     
    }

We will get an array {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10}

a
almightyGOSU

Code: 

List<String> duplicatList = new ArrayList<String>();
duplicatList = Arrays.asList("AA","BB","CC","DD","DD","EE","AA","FF");
//above AA and DD are duplicate
Set<String> uniqueList = new HashSet<String>(duplicatList);
duplicatList = new ArrayList<String>(uniqueList); //let GC will doing free memory
System.out.println("Removed Duplicate : "+duplicatList);

Note: Definitely, there will be memory overhead.

H
Hardip 
ArrayList<String> city=new ArrayList<String>();
city.add("rajkot");
city.add("gondal");
city.add("rajkot");
city.add("gova");
city.add("baroda");
city.add("morbi");
city.add("gova");

HashSet<String> hashSet = new HashSet<String>();
hashSet.addAll(city);
city.clear();
city.addAll(hashSet);
Toast.makeText(getActivity(),"" + city.toString(),Toast.LENGTH_SHORT).show();
H
HamidReza

you can use nested loop in follow : 

ArrayList<Class1> l1 = new ArrayList<Class1>();
ArrayList<Class1> l2 = new ArrayList<Class1>();

        Iterator iterator1 = l1.iterator();
        boolean repeated = false;

        while (iterator1.hasNext())
        {
            Class1 c1 = (Class1) iterator1.next();
            for (Class1 _c: l2) {
                if(_c.getId() == c1.getId())
                    repeated = true;
            }
            if(!repeated)
                l2.add(c1);
        }
Perfect - just missing "repeated = false;" in the internal loop after the "if(!repeated) l2.add(c1);" otherwise it return a short list
u
user1912383

LinkedHashSet will do the trick. 

String[] arr2 = {"5","1","2","3","3","4","1","2"};
Set<String> set = new LinkedHashSet<String>(Arrays.asList(arr2));
for(String s1 : set)
    System.out.println(s1);

System.out.println( "------------------------" );
String[] arr3 = set.toArray(new String[0]);
for(int i = 0; i < arr3.length; i++)
     System.out.println(arr3[i].toString());

//output: 5,1,2,3,4

s
siva 
        List<String> result = new ArrayList<String>();
        Set<String> set = new LinkedHashSet<String>();
        String s = "ravi is a good!boy. But ravi is very nasty fellow.";
        StringTokenizer st = new StringTokenizer(s, " ,. ,!");
        while (st.hasMoreTokens()) {
            result.add(st.nextToken());
        }
         System.out.println(result);
         set.addAll(result);
        result.clear();
        result.addAll(set);
        System.out.println(result);

output:
[ravi, is, a, good, boy, But, ravi, is, very, nasty, fellow]
[ravi, is, a, good, boy, But, very, nasty, fellow]
G
Gujjula Ramesh Reddy

This is used for your Custom Objects list 

   public List<Contact> removeDuplicates(List<Contact> list) {
    // Set set1 = new LinkedHashSet(list);
    Set set = new TreeSet(new Comparator() {

        @Override
        public int compare(Object o1, Object o2) {
            if (((Contact) o1).getId().equalsIgnoreCase(((Contact) o2).getId()) /*&&
                    ((Contact)o1).getName().equalsIgnoreCase(((Contact)o2).getName())*/) {
                return 0;
            }
            return 1;
        }
    });
    set.addAll(list);

    final List newList = new ArrayList(set);
    return newList;
}
Y
Yoon5oo

As said before, you should use a class implementing the Set interface instead of List to be sure of the unicity of elements. If you have to keep the order of elements, the SortedSet interface can then be used; the TreeSet class implements that interface.

P
Pigueiras 
import java.util.*;
class RemoveDupFrmString
{
    public static void main(String[] args)
    {

        String s="appsc";

        Set<Character> unique = new LinkedHashSet<Character> ();

        for(char c : s.toCharArray()) {

            System.out.println(unique.add(c));
        }
        for(char dis:unique){
            System.out.println(dis);
        }


    }
}
H
Harsha 
public Set<Object> findDuplicates(List<Object> list) {
        Set<Object> items = new HashSet<Object>();
        Set<Object> duplicates = new HashSet<Object>();
        for (Object item : list) {
            if (items.contains(item)) {
                duplicates.add(item);
                } else { 
                    items.add(item);
                    } 
            } 
        return duplicates;
        }
S
SparkOn 
    ArrayList<String> list = new ArrayList<String>();
    HashSet<String> unique = new LinkedHashSet<String>();
    HashSet<String> dup = new LinkedHashSet<String>();
    boolean b = false;
    list.add("Hello");
    list.add("Hello");
    list.add("how");
    list.add("are");
    list.add("u");
    list.add("u");

    for(Iterator iterator= list.iterator();iterator.hasNext();)
    {
        String value = (String)iterator.next();
        System.out.println(value);

        if(b==unique.add(value))
            dup.add(value);
        else
            unique.add(value);


    }
    System.out.println(unique);
    System.out.println(dup);
T
Thananjayan N

If you want to remove duplicates from ArrayList means find the below logic, 

public static Object[] removeDuplicate(Object[] inputArray)
{
    long startTime = System.nanoTime();
    int totalSize = inputArray.length;
    Object[] resultArray = new Object[totalSize];
    int newSize = 0;
    for(int i=0; i<totalSize; i++)
    {
        Object value = inputArray[i];
        if(value == null)
        {
            continue;
        }

        for(int j=i+1; j<totalSize; j++)
        {
            if(value.equals(inputArray[j]))
            {
                inputArray[j] = null;
            }
        }
        resultArray[newSize++] = value;
    }

    long endTime = System.nanoTime()-startTime;
    System.out.println("Total Time-B:"+endTime);
    return resultArray;
}
Why would you post a quadratic solution to a question that already has 2-year-old linear and log-linear solutions, that are also simpler?

关注公众号,不定期副业成功案例分享
Follow WeChat

Success story sharing

Want to stay one step ahead of the latest teleworks?

Subscribe Now

HuntsBot,a one-stop outsourcing task, remote job, product ideas sharing and subscription platform, which supports DingTalk, Lark, WeCom, Email and Telegram robot subscription. The platform will push outsourcing task requirements, remote work opportunities, product ideas to every subscribed user with timely, stable and reliable.

简体中文 English

Platform

Support

Contact US

Any questions or suggestions during use, you can contact us in the following ways:

Email: huntsbot@xinbeitime.com

Copyright© 2022-2023 www.huntsbot.com 浙ICP备2022000860号-4 All Rights Reserved.