The 18 ways removing duplicates for Java Array or List
更多语言去重算法：github.com/microwind/a…
- Traverse all members, compare the current item with the left item one by one, if the value is the same and the index is the same, it means unique, and the others are considered to be duplicates and ignored.
static int[] unique1(int arr[]) {
int newArr[] = new int[arr.length];
int x = 0;
for (int i = 0; i < arr.length; i++) {
for (int j = 0; j <= i; j++) {
if (arr[i] == arr[j]) {
if (i == j) {
newArr[x] = arr[i];
x++;
}
break;
}
}
}
int result[] = Arrays.copyOf(newArr, x);
return result;
}
- First convert the Array to a List, use the
indexOfmethod of List to find the index, when the index matches, it is unique and add it to the new List.
static Integer[] unique2(Integer arr[]) {
int x = 0;
List<Integer> list = new ArrayList<>(Arrays.asList(arr));
int l = list.size();
for (int i = 0; i < l; i++) {
if (list.indexOf(arr[i]) == i) {
list.add(arr[i]);
x++;
}
}
// get the unique item
Integer[] result = new Integer[x];
return list.subList(list.size() - x,list.size()).toArray(result);
}
- Remove duplicate items from the original List. Traverse from end to front, compare current with the previous items one by one, if the value is the same and the index too, then remove the current item.
static Integer[] unique3(Integer arr[]) {
List<Integer> list = new ArrayList<>(Arrays.asList(arr));
int l = list.size();
while (l-- > 0) {
int i = l;
while (i-- > 0) {
if (list.get(l).equals(list.get(i))) {
list.remove(l);
break;
}
}
}
return list.toArray(new Integer[list.size()]);
}
- Remove duplicate items from the original List. Traverse from front to end, compare with the previous items one by one, if the value is the same and the index is the same, remove the previous item.
static Integer[] unique4(Integer arr[]) {
List<Integer> list = new ArrayList<>(Arrays.asList(arr));
int l = list.size();
for (int i = 1; i < l; i++) {
for (int j = 0; j < i; j++) {
if (list.get(i).equals(list.get(j))) {
list.remove(i);
i--;
l--;
break;
}
}
}
return list.toArray(new Integer[list.size()]);
}
- Remove duplicate items from the original List. Traverse from front to end, compare with the following items one by one, and remove the current item if the value is the same and the index is the same.
static Integer[] unique5(Integer arr[]) {
List<Integer> list = new ArrayList<>(Arrays.asList(arr));
int l = list.size();
for (int i = 0; i < l; i++) {
for (int j = i + 1; j < l; j++) {
if (list.get(i).equals(list.get(j))) {
list.remove(j);
i--;
l--;
break;
}
}
}
return list.toArray(new Integer[list.size()]);
}
- Use HashMap attribute special to achieve deduplication.
static Integer[] unique6(Integer arr[]) {
Map<Object, Integer> map = new HashMap<Object, Integer>();
for (Integer item : arr) {
if (map.containsKey(item)) {
continue;
}
map.put(item, item);
}
List<Integer> list = new ArrayList<>(map.values());
return list.toArray(new Integer[list.size()]);
}
- Use the expression to filter out the unqualified ones. Need to use an external list to store unique items.
static List<Integer> unique7newArr = new ArrayList<>();
static boolean unique7contains(Integer item) {
if (unique7newArr.indexOf(item) < 0) {
unique7newArr.add(item);
return true;
}
return false;
}
static Integer[] unique7(Integer arr[]) {
List<Integer> list = new ArrayList<>(Arrays.asList(arr));
return list.stream().filter(UniqueArray::unique7contains).collect(Collectors.toList())
.toArray(new Integer[UniqueArray.unique7newArr.size()]);
}
- Use the HashSet data structure to remove duplicate items. Out of order and out of sync.
static Integer[] unique8(Integer arr[]) {
System.out.print("covert to steam first then to set: ");
Arrays.asList(arr).stream().collect(Collectors.toSet())
.forEach(System.out::print);
System.out.println("\ndirectly convert to set:");
Set<Integer> set = new HashSet<>(Arrays.asList(arr));
return new ArrayList<>(set).toArray(new Integer[set.size()]);
}
- Use LinkedHashSet data structure to remove duplicate items.
static Integer[] unique9(Integer arr[]) {
Set<Integer> linkedHashSet = new LinkedHashSet<>(Arrays.asList(arr));
return new ArrayList<>(linkedHashSet).toArray(new Integer[linkedHashSet.size()]);
}
- Use TreeSet data structure to remove duplicate items.
static Integer[] unique10(Integer arr[]) {
Set<Integer> treeSet = new TreeSet<>(Arrays.asList(arr)).descendingSet();
return new ArrayList<>(treeSet).toArray(new Integer[treeSet.size()]);
}
11.Sort first, traverse from end to front, compare the current item with the front item, and remove the current item if it is repeated.
static Integer[] unique11(Integer arr[]) {
List<Integer> list = new ArrayList<>(Arrays.asList(arr));
Collections.sort(list);
for (int l = list.size() - 1; l > 0; l--) {
if (list.get(l).equals(list.get(l - 1))) {
list.remove(l);
}
}
return new ArrayList<>(list).toArray(new Integer[list.size()]);
}
- Sort in advance, traverse from front to end, compare the current item with the next item, and remove the current item if it is repeated.
static Integer[] unique12(Integer arr[]) {
List<Integer> list = new ArrayList<>(Arrays.asList(arr));
Collections.sort(list, Collections.reverseOrder());
int l = list.size() - 1;
for (int i = 0; i < l; i++) {
if (list.get(i).equals(list.get(i + 1))) {
list.remove(i);
i--;
l--;
}
}
return new ArrayList<>(list).toArray(new Integer[list.size()]);
}
- convert List to Stream，use distinct method to make unique.
static Integer[] unique13(Integer arr[]) {
List<Integer> list = new ArrayList<>(Arrays.asList(arr));
list = list.stream().distinct().collect(Collectors.toList());
return new ArrayList<>(list).toArray(new Integer[list.size()]);
}
- Double circle loop from right to left, compare current item with left one by one. if meet same item go to the next, comparison continue. Non-duplicate items are unique and appended to the new array.
static Integer[] unique14(Integer arr[]) {
int len = arr.length;
Integer[] result = new Integer[len];
int x = len;
for (int i = len - 1; i >= 0; i--) {
for (int j = i - 1; j >= 0; j--) {
if (arr[i].equals(arr[j])) {
i--;
j = i;
}
}
result[--x] = arr[i];
}
return Arrays.copyOfRange(result, x, len);
}
- Use Iterator to reverse the List,Of the new List has no the current item to add it.
static Integer[] unique15(Integer arr[]) {
List<Integer> list = new ArrayList<>(Arrays.asList(arr));
List<Integer> result = new ArrayList<>();
Iterator<Integer> it = list.iterator();
while (it.hasNext()) {
Integer item = it.next();
if (!result.contains(item)) {
result.add(item);
}
}
return new ArrayList<>(result).toArray(new Integer[result.size()]);
}
- Use recursive calls to repeat. The recursion is called one by one from end to front, and ends when the length is 1. When the latter item has the same index as the front item, there is a duplication, then delete the current item. Equivalent to using self-call to replace the loop.
static Integer[] uniqueRecursion1(Integer arr[], int len, List<Integer> result) {
int last = len - 1;
Integer lastItem = arr[last];
int l = last;
boolean isRepeat = false;
if (len <= 1) {
result.add(0, lastItem);
return new ArrayList<>(result).toArray(new Integer[result.size()]);
}
while (l-- > 0) {
if (lastItem.equals(arr[l])) {
isRepeat = true;
break;
}
}
// add the unique to list
if (!isRepeat) {
result.add(0, lastItem);
}
return uniqueRecursion1(arr, len - 1, result);
}
- The other way like above. Use recursive calls the repeat.
static List<Integer> uniqueRecursion2(List<Integer> arr, int len) {
if (len <= 1) {
System.out.println("last arr:" + arr);
return arr;
}
int last = len - 1;
int l = last - 1;
boolean isRepeat = false;
Integer lastItem = arr.get(last);
while (l >= 0) {
if (lastItem.equals(arr.get(l))) {
isRepeat = true;
break;
}
l--;
}
// join the unique item to result
List<Integer> result = new ArrayList<>();
arr.remove(last);
if (!isRepeat) {
result.add(lastItem);
}
return Stream.concat(uniqueRecursion2(arr, len - 1).stream(), result.stream()).collect(Collectors.toList());
}
- Double loop, compare the items on the left with the current items one by one. If the values are equal, it will jump out of the loop to compare the index, and if the index are the same, it will be appended to the new array. This is slightly different from the first scheme.
static Integer[] unique18(Integer arr[]) {
Integer newArr[] = new Integer[arr.length];
int x = 0;
for (int i = 0; i < arr.length; i++) {
int j = 0;;
for (; j < i; j++) {
if (arr[i].equals(arr[j])) {
break;
}
}
if (i == j) {
newArr[x] = arr[i];
x++;
}
}
return Arrays.copyOf(newArr, x);
}
more: github.com/microwind/a…
import java.util.*;
import java.util.stream.Collectors;
import java.util.stream.Stream;
public class UniqueArray {
public static void main(final String args[]) {
int arr1[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
int[] result;
long startTime;
// 1.
System.out.println("unique1 start:" + Arrays.toString(arr1));
startTime = System.currentTimeMillis();
result = UniqueArray.unique1(arr1);
System.out.println("unique1 result:" + Arrays.toString(result));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 2.
Integer arr2[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique2 start:" + Arrays.toString(arr2));
startTime = System.currentTimeMillis();
Integer result2[] = UniqueArray.unique2(arr2);
System.out.println("unique2 result:" + Arrays.toString(result2));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 3.
Integer arr3[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique3 start:" + Arrays.toString(arr2));
startTime = System.currentTimeMillis();
Integer result3[] = UniqueArray.unique3(arr3);
System.out.println("unique3 result:" + Arrays.toString(result3));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 4.
Integer arr4[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique4 start:" + Arrays.toString(arr4));
startTime = System.currentTimeMillis();
Integer result4[] = UniqueArray.unique4(arr4);
System.out.println("unique4 result:" + Arrays.toString(result4));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 5.
Integer arr5[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique5 start:" + Arrays.toString(arr5));
startTime = System.currentTimeMillis();
Integer result5[] = UniqueArray.unique5(arr5);
System.out.println("unique5 result:" + Arrays.toString(result5));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 6.
Integer arr6[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique6 start:" + Arrays.toString(arr6));
startTime = System.currentTimeMillis();
Integer result6[] = UniqueArray.unique6(arr6);
System.out.println("unique6 result:" + Arrays.toString(result6));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 7.
Integer arr7[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique7 start:" + Arrays.toString(arr7));
startTime = System.currentTimeMillis();
Integer result7[] = UniqueArray.unique7(arr7);
System.out.println("unique7 result:" + Arrays.toString(result7));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 8.
Integer arr8[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique8 start:" + Arrays.toString(arr8));
startTime = System.currentTimeMillis();
Integer result8[] = UniqueArray.unique8(arr8);
System.out.println("unique8 result:" + Arrays.toString(result8));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 9.
Integer arr9[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique9 start:" + Arrays.toString(arr9));
startTime = System.currentTimeMillis();
Integer result9[] = UniqueArray.unique9(arr9);
System.out.println("unique9 result:" + Arrays.toString(result9));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 10.
Integer arr10[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique10 start:" + Arrays.toString(arr10));
startTime = System.currentTimeMillis();
Integer result10[] = UniqueArray.unique10(arr10);
System.out.println("unique10 result:" + Arrays.toString(result10));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 11.
Integer arr11[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique11 start:" + Arrays.toString(arr11));
startTime = System.currentTimeMillis();
Integer result11[] = UniqueArray.unique11(arr11);
System.out.println("unique11 result:" + Arrays.toString(result11));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 12.
Integer arr12[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique12 start:" + Arrays.toString(arr12));
startTime = System.currentTimeMillis();
Integer result12[] = UniqueArray.unique12(arr12);
System.out.println("unique12 result:" + Arrays.toString(result12));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 13.
Integer arr13[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique13 start:" + Arrays.toString(arr13));
startTime = System.currentTimeMillis();
Integer result13[] = UniqueArray.unique13(arr13);
System.out.println("unique13 result:" + Arrays.toString(result13));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 14.
Integer arr14[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique14 start:" + Arrays.toString(arr14));
startTime = System.currentTimeMillis();
Integer result14[] = UniqueArray.unique14(arr14);
System.out.println("unique14 result:" + Arrays.toString(result14));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 15.
Integer arr15[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique15 start:" + Arrays.toString(arr15));
startTime = System.currentTimeMillis();
Integer result15[] = UniqueArray.unique15(arr15);
System.out.println("unique15 result:" + Arrays.toString(result15));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 16.
Integer arr16[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("uniqueRecursion1 start:" + Arrays.toString(arr16));
startTime = System.currentTimeMillis();
Integer result16[] = UniqueArray.uniqueRecursion1(arr16, arr16.length, new ArrayList<>());
System.out.println("uniqueRecursion1 result:" + Arrays.toString(result16));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 17.
Integer arr17[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("uniqueRecursion2 start:" + Arrays.toString(arr17));
startTime = System.currentTimeMillis();
List<Integer> result17 = UniqueArray.uniqueRecursion2(new ArrayList<>(Arrays.asList(arr17)), arr17.length);
System.out.println("uniqueRecursion2 result:" + result17);
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
// 18.
Integer arr18[] = { 1, 3, -1, 1, 2, 2, 4, 2, 2, -1 };
System.out.println("unique18 start:" + Arrays.toString(arr18));
startTime = System.currentTimeMillis();
Integer result18[] = UniqueArray.unique18(arr18);
System.out.println("unique18 result:" + Arrays.toString(result18));
System.out.println("\r\ntime:" + (System.currentTimeMillis() - startTime) + " ms.");
}
result：
jarrys-MacBook-Pro:unique jarry$ javac UniqueArray.java
jarrys-MacBook-Pro:unique jarry$ java UniqueArray
unique1 start:[1, 3, -1, 1, 2, 2, 4, 2, 2, -1]
unique1 result:[1, 3, -1, 2, 4]
